c6feb78653
more failures
2022-09-27 19:02:17 +02:00
b9003fb91d
add a list of urls not to try to download (since they fail or have bad checksums)
2022-09-27 18:56:08 +02:00
5e427897be
use git-kv on store to dump and restore the git commit
2022-09-27 17:24:37 +02:00
83d494c433
Merge pull request 'read the data in chunks and send them chunk-wise' ( #16 ) from partial-read into main
...
Reviewed-on: https://git.robur.io/robur/opam-mirror/pulls/16
2022-09-27 14:54:36 +00:00
8bdfb19e6f
use read_partial in init as well for checksum computation
2022-09-27 16:25:57 +02:00
007842f3d3
flush and wait
2022-09-27 12:22:29 +02:00
938da1a211
read the data in chunks and send them chunk-wise
2022-09-27 10:46:14 +02:00
c81ba101f9
Merge pull request 'Hash cache' ( #14 ) from partition' into main
...
Reviewed-on: https://git.robur.io/robur/opam-mirror/pulls/14
2022-09-27 08:05:27 +00:00
9bb066cd56
minor cleanups
2022-09-26 21:51:42 +02:00
b26e23c462
add --verify to force verification of the on-disk data
...
by default, do not compute hashes unless requested or required
2022-09-26 21:49:47 +02:00
24dda11659
Hash cache
...
The block devie is partitioned into a tar rw filesystem and two hash map
cache partitions. The latter store the computes maps from {md5,sha512}
checksums to sha256 checksums to speed up the boot process.
2022-09-26 20:17:45 +02:00
e22047249f
log git interaction
2022-09-26 18:33:29 +02:00
329b78c923
fast path if nothing changed upstream
2022-09-26 15:03:35 +02:00
dd3b2f168c
minor README nit
2022-09-26 14:11:59 +02:00
0cf002e0cd
update git repo and fetch archives hourly
2022-09-26 14:11:23 +02:00
44d737887a
use git_kv
...
Co-Authored-By: Hannes Mehnert <hannes@mehnert.org>
2022-09-26 13:46:47 +02:00
9b7e9e5485
use the gc every now and then
2022-09-26 13:46:23 +02:00
069643465f
allow to specify the number of parallel downloads as boot parameter
2022-09-25 23:13:52 +02:00
b8a369dac8
Fix typo
2022-09-22 13:41:19 +02:00
8dbd3ed663
remove key-hex, it is the default now
2022-09-22 13:20:11 +02:00
7ec50fe588
comment out non-working code for now
2022-09-21 11:21:06 +02:00
b6218b979b
attempt to serialize git commit (does not compile)
2022-09-07 10:33:21 +02:00
99dcf500c1
once the http fetch is done, close the flow ( #5 )
2022-09-07 09:09:15 +02:00
023fb4283d
use tar and extend the block device
2022-09-07 09:08:45 +02:00
640451bfaa
avoid attempting to download the big [tm] archive
2022-09-07 08:40:40 +02:00
6f9d5706cc
WIP: use tar
2022-09-06 09:55:00 +02:00
ef2ec2e946
add a hook_url to conduct a git pull
2022-09-04 14:52:42 +02:00
3d1bb6c153
unikernel: remove unused module type DNS
2022-09-04 13:25:33 +02:00
7d4387807b
minor
2022-09-04 10:28:38 +02:00
f62f2a09ce
in addition to retrieve stuff, also serve a repository:
...
- archive cache
- repository (index.tar.gz) dynamically created from tarball
- repo file (generated including the commit id and repository URL)
2022-09-04 10:01:45 +02:00
c665bb23fb
Merge pull request 'Fix a leak of fd from h2 - we must shutdown the connection to really close the fd' ( #6 ) from fix-h2-leak into main
...
Reviewed-on: https://git.robur.io/robur/opam-mirror/pulls/6
2022-09-01 07:53:10 +00:00
d03cd65dcd
Fix a leak of fd from h2 - we must shutdown the connection to really close the fd
2022-09-01 09:50:53 +02:00
3aca170154
use a direct_kv_rw (unix only)
2022-09-01 09:32:29 +02:00
a3fdc527d2
new flag: key-hex to specify whether the KV-keys should be hex-encoded or raw binary
2022-09-01 09:31:53 +02:00
6d51eb707b
minor: silence warnings
2022-09-01 09:30:46 +02:00
a1e4b71eb9
unikernel: new flag --check to only check data (and hashes) on disk
2022-09-01 09:30:11 +02:00
c16f2288ed
Use Status.is_redirection instead of matching on the polymorphic variant
...
The reason behind this is that the new HTTP code 308 (permanent redirect)
is not recognized by HTTP/AF, but already used (e.g. by alt-ergo.ocamlpro.com).
2022-09-01 09:27:03 +02:00
86c961f5cf
less debug
2022-08-31 16:25:04 +02:00
bf29496274
Merge pull request 'Limit the number of concurrent downloads' ( #4 ) from download-pool into main
...
Reviewed-on: https://git.robur.io/robur/opam-mirror/pulls/4
2022-08-30 16:42:40 +00:00
030e6d8ab7
Limit the number of concurrent downloads
...
At the moment this is hardcoded to 20 concurrent downloads.
2022-08-30 16:51:55 +02:00
236e09b7e0
First PoC about a native HTTP client on Mirage with mimic/paf/httpaf/h2
2022-08-30 16:12:55 +02:00
5d86452f5f
(a) paf 0.1.0 is released now, which works fine (no pins needed)
...
(b) use 16 as program_block_size, not 512 - this results in a working unikernel
(c) report data pushed into the map
2022-08-30 11:35:28 +02:00
c3d66b96b1
use chamelon to store
2022-08-26 17:23:46 +02:00
cd3294ebe3
only download stuff that is not present
2022-08-26 15:34:20 +02:00
a61f944d4f
read and dump to a kv, taking csums into account
2022-08-26 15:18:02 +02:00
6fe03ce867
git clone and extract urls & checksums
2022-08-25 22:47:46 +02:00
788f58d3cf
minor: no console
2022-08-25 15:13:42 +02:00
8b5bfdd789
initial commit
2022-08-25 14:57:03 +02:00
