6490801ce1
Update dependencies, refactor
...
- Tar.3.0.0 was released! So we can drop the pin and depend on it
directly.
- Use digestif in favor of the bygone mirage-crypto hash
- Move most of the hashing logic into archive_checksum.ml
- When checking the checksum we should check the checksum chosen, too. I
believe this was a bug that was hidden due to opam packages always
using one of md5, sha256 or sha512 for checksums.
2024-10-03 14:37:37 +02:00
8fc8f1c62d
Update the unikernel with the new version of tar
2024-07-31 14:54:26 +02:00
c2ffbdb891
Update to mirage.4.6.0
2024-07-15 12:22:57 +02:00
1ebf370a4f
compiles now
2023-05-02 16:13:43 +02:00
Robur
fd8ce3be03
more wip
2023-05-02 12:03:25 +00:00
ebeadf69d8
Skip unavailable packages
2023-02-08 16:30:56 +01:00
adf6564385
Print error and exit if we can't get commit id
2023-02-08 16:30:30 +01:00
8c7e71127f
Try to remove files before marking for deletion
...
You know, it might just work...
2023-02-08 16:29:55 +01:00
1a6c6b8f9d
Return error on bad checksum when finalizing write
2023-01-31 10:27:25 +01:00
8e326ecbc5
WIP partial writes
...
If possible, downloads are streamed to disk in the /pending/ directory
in the tar filesystem. If the download is successful and the checksums
match the file is renamed to its sha256 hash. Otherwise, it is moved
under /to-delete/ so it can be deleted by an operator.
Before downloading we check if it has been downloaded before, but we
need to check as well if it is being downloaded (in /pending/) or if we
unsuccessfully downloaded it before (e.g. failed checksum, stored in
/to-delete/).
It is not very elegant code, and it could do with a thorough review or
rewrite.
2023-01-25 11:34:31 +01:00
b76f2997f5
Migrate to mirage-kv 6
2023-01-23 09:13:42 +01:00
b0eb816a68
Log before and after restoring/fetching git state
...
This may take quite a while, and the application is unresponsive
meanwhile.
2022-11-17 12:12:15 +01:00
c6e8c6cf77
update for the released http-mirage-client
2022-11-02 22:19:03 +01:00
d143e9b766
as suggested by @reynir : verify -> verify_sha256
2022-10-28 14:58:58 +02:00
1ff5c7e1b6
if --verify is passed, don't use the md5s/sha512s from disk, instead re-create them
2022-10-28 13:57:30 +02:00
0d5745b340
Add a boot argument to ignore the local git state
2022-10-26 18:32:43 +02:00
7d1c9b0814
Fix the compilation of opam-mirror with lastest version of packages
2022-10-21 15:56:20 +02:00
5f761b38fb
Upgrade opam-mirror with last release of git and new version of git-kv
2022-10-20 16:09:05 +02:00
e177cb156d
adapt to new paf
2022-09-29 14:40:50 +02:00
3e946395aa
Rename partition devices so they are shadowed
2022-09-28 11:18:17 +02:00
b9003fb91d
add a list of urls not to try to download (since they fail or have bad checksums)
2022-09-27 18:56:08 +02:00
5e427897be
use git-kv on store to dump and restore the git commit
2022-09-27 17:24:37 +02:00
8bdfb19e6f
use read_partial in init as well for checksum computation
2022-09-27 16:25:57 +02:00
007842f3d3
flush and wait
2022-09-27 12:22:29 +02:00
938da1a211
read the data in chunks and send them chunk-wise
2022-09-27 10:46:14 +02:00
c81ba101f9
Merge pull request 'Hash cache' ( #14 ) from partition' into main
...
Reviewed-on: https://git.robur.io/robur/opam-mirror/pulls/14
2022-09-27 08:05:27 +00:00
9bb066cd56
minor cleanups
2022-09-26 21:51:42 +02:00
b26e23c462
add --verify to force verification of the on-disk data
...
by default, do not compute hashes unless requested or required
2022-09-26 21:49:47 +02:00
24dda11659
Hash cache
...
The block devie is partitioned into a tar rw filesystem and two hash map
cache partitions. The latter store the computes maps from {md5,sha512}
checksums to sha256 checksums to speed up the boot process.
2022-09-26 20:17:45 +02:00
e22047249f
log git interaction
2022-09-26 18:33:29 +02:00
329b78c923
fast path if nothing changed upstream
2022-09-26 15:03:35 +02:00
0cf002e0cd
update git repo and fetch archives hourly
2022-09-26 14:11:23 +02:00
44d737887a
use git_kv
...
Co-Authored-By: Hannes Mehnert <hannes@mehnert.org>
2022-09-26 13:46:47 +02:00
9b7e9e5485
use the gc every now and then
2022-09-26 13:46:23 +02:00
069643465f
allow to specify the number of parallel downloads as boot parameter
2022-09-25 23:13:52 +02:00
b8a369dac8
Fix typo
2022-09-22 13:41:19 +02:00
8dbd3ed663
remove key-hex, it is the default now
2022-09-22 13:20:11 +02:00
7ec50fe588
comment out non-working code for now
2022-09-21 11:21:06 +02:00
b6218b979b
attempt to serialize git commit (does not compile)
2022-09-07 10:33:21 +02:00
023fb4283d
use tar and extend the block device
2022-09-07 09:08:45 +02:00
640451bfaa
avoid attempting to download the big [tm] archive
2022-09-07 08:40:40 +02:00
6f9d5706cc
WIP: use tar
2022-09-06 09:55:00 +02:00
ef2ec2e946
add a hook_url to conduct a git pull
2022-09-04 14:52:42 +02:00
3d1bb6c153
unikernel: remove unused module type DNS
2022-09-04 13:25:33 +02:00
7d4387807b
minor
2022-09-04 10:28:38 +02:00
f62f2a09ce
in addition to retrieve stuff, also serve a repository:
...
- archive cache
- repository (index.tar.gz) dynamically created from tarball
- repo file (generated including the commit id and repository URL)
2022-09-04 10:01:45 +02:00
a3fdc527d2
new flag: key-hex to specify whether the KV-keys should be hex-encoded or raw binary
2022-09-01 09:31:53 +02:00
6d51eb707b
minor: silence warnings
2022-09-01 09:30:46 +02:00
a1e4b71eb9
unikernel: new flag --check to only check data (and hashes) on disk
2022-09-01 09:30:11 +02:00
86c961f5cf
less debug
2022-08-31 16:25:04 +02:00
