Use swapfs

reynir commented

2024-11-01 13:47:57 +00:00

Owner

This breaks compatibility with previous disk layout, and there is no upgrade path /o\

The swapfs could very well be used for downloads of files with a known size - it happens that the download is interrupted or the checksum is incorrect. In that case, we then have a file in the tar filesystem that is attempted deleted, and with lots of concurrent download jobs that means it is tomb stoned as /to-delete/blablah /o\ so I may do that at some point.

This breaks compatibility with previous disk layout, and there is no upgrade path /o\ The swapfs could very well be used for downloads of files with a known size - it happens that the download is interrupted or the checksum is incorrect. In that case, we then have a file in the tar filesystem that is attempted deleted, and with lots of concurrent download jobs that means it is tomb stoned as `/to-delete/blablah` /o\ so I may do that at some point.

reynir added 1 commit 2024-11-01 13:47:57 +00:00

Use swapfs 456340562d

hannes added 10 commits 2024-11-04 17:44:03 +00:00

always use swap, remove the pending / to_delete stuff e59f02a16f

remove bad archive list 7689397ac3

drop superfluous 'unknown' f48cc19fc4

restore checksum failure error 4bec3bfbd8

first write to a temporary filename, and rename later 2312092e42

stream git contents, also make the tarball and the find_urls in one go a9b8f18192

initialization: potentially rename bad data 1e35bfefbd

tweaks a47193f147

record and preserve opam file parsing issues 9c50538877

group download failures by error 8ba4cfae00

hannes commented

2024-11-04 17:46:28 +00:00

Owner

I added some commits... This includes: always use swapfs (even for fixed size downloads).

There's also more detailed error reporting.

And a TODO, would be nice to have a "set_stream" in Tar so we can avoid a temporary filename (and always ensure that we only write good data to the tar) -- but due to potential disk failures, we need to have a path to skip/remove broken data anyways.

I added some commits... This includes: always use swapfs (even for fixed size downloads). There's also more detailed error reporting. And a TODO, would be nice to have a "set_stream" in Tar so we can avoid a temporary filename (and always ensure that we only write good data to the tar) -- but due to potential disk failures, we need to have a path to skip/remove broken data anyways.

hannes commented

2024-11-04 18:42:32 +00:00

Owner

With this PR, even when using a parallel_download of 1, and exiting the unikernel after the first 14 downloads, I get only a few correct checksums:

$ for x in $(ls); do sha256 $x ; done
SHA256 (0f92632727e0d42cbb12208b5996a697b7a59ebb2ae26db4afdbf4c09a4e36f1) = 5742dcef8c02051cd05af5a8f8248d01783b3fd6207afcd902a27a838b5f8ca3
SHA256 (0fd6594f30d4e2bea97350a7906ecc2528c37e8fdd32588818a162aacc409688) = b4be65b20c22a47417495f7c9d40740633480ef3e9cdcb5eeafb223ae18cefbf
SHA256 (1f3e31c5ba9e0c2d453442677e1935e13da499036770eff40c105e631b2304f6) = 21bf2785f68a35960b64e37e136970174609a90e3d25f4a8b4f83a688516d024
SHA256 (2fd8428323efeafe1f0a92aeaf98242f4668784567a25eac230675451653b708) = b90e693792e47622ab42ae93fcebbdbfc1251973a6a85fc4e07239a72f285a85
SHA256 (333ec32cf6db9f1a54b2a6cb5af97ec0e233e355a6b3d73f1963ed0f7d49be37) = 0e1158b24c47cf6743ad4168d62f0269b75260b0adbfdd73eb09b9ce8d7c82f0
SHA256 (5a6cd4349c144653be19a1ba4c254bbf626bdfd97c54d1c13ba63e396006eeac) = 210661183206a064208b3fc347287f904965c1ace6937ee230f8351bb3387d80
SHA256 (5c2445f99c567cf4c4521cd9ec8873f8788a4c7ce7a349dbc91d1bace373df80) = fb903771238383b516e78189fc382d48b8d8b2e21232ad8a87f02d039060ebfa
SHA256 (5fe64c7db492eac1404a82eeb6f83341df3f3793ddb604e575e90d24a3de64dc) = fcb71b978e2ea06225abd3e725228910b9eeb52e432d331c88049bd7df053f69
SHA256 (7f1906900272125315833b9f0a6abc3c5af7b836d604fdb10a98a9079c1b99f9) = f4350476aa84805a2efe70c89849ad8ee1ad451c9678ddb28eda1306f686364a
SHA256 (8f67a6dfad5be14636c80e4b4c4d7c42bd921009b33261b017fa54d07ad58910) = 8f67a6dfad5be14636c80e4b4c4d7c42bd921009b33261b017fa54d07ad58910
SHA256 (c77e17a2af1d2811f8afcdbc0faf7d6a3acc324730e4a36317e753d0fc9aa3fc) = f80171efe5b86e53df2a7254f9fa335ac1d2eca3c1ba9210bdd1cb50d6ffa4a5
SHA256 (d51a9833acf265d475408cde04e9beb75f950355580b9c73292ffbfbebf280eb) = a8dabbbae9352d4faa5ceb86a1ddf453d95dd3891c2ae4d52b56ecf8aa1de280
SHA256 (d8f42c764ad73c2370713d5f03ceb255bdd96518130a95955606c9ad511823bf) = d8f42c764ad73c2370713d5f03ceb255bdd96518130a95955606c9ad511823bf
SHA256 (f5f24f0af60ff5b4a7d054e95bc677befbe708859fafdd5b214a47ed4d71d452) = 2e0fe3cc142b0926ab4143b3fd72566ae7b9ba9a3d96e3cdbc14966b58513561

I'm lost where the issue is... since the "write to tar" should only happen when the checksum is correct -- so maybe the swapfs is confused / has somewhere an issue?

As mentioned, this is with a single parallel download (which doesn't prevent that there can be multiple handles used at the swap at the same time).

With this PR, even when using a parallel_download of 1, and exiting the unikernel after the first 14 downloads, I get only a few correct checksums: ``` $ for x in $(ls); do sha256 $x ; done SHA256 (0f92632727e0d42cbb12208b5996a697b7a59ebb2ae26db4afdbf4c09a4e36f1) = 5742dcef8c02051cd05af5a8f8248d01783b3fd6207afcd902a27a838b5f8ca3 SHA256 (0fd6594f30d4e2bea97350a7906ecc2528c37e8fdd32588818a162aacc409688) = b4be65b20c22a47417495f7c9d40740633480ef3e9cdcb5eeafb223ae18cefbf SHA256 (1f3e31c5ba9e0c2d453442677e1935e13da499036770eff40c105e631b2304f6) = 21bf2785f68a35960b64e37e136970174609a90e3d25f4a8b4f83a688516d024 SHA256 (2fd8428323efeafe1f0a92aeaf98242f4668784567a25eac230675451653b708) = b90e693792e47622ab42ae93fcebbdbfc1251973a6a85fc4e07239a72f285a85 SHA256 (333ec32cf6db9f1a54b2a6cb5af97ec0e233e355a6b3d73f1963ed0f7d49be37) = 0e1158b24c47cf6743ad4168d62f0269b75260b0adbfdd73eb09b9ce8d7c82f0 SHA256 (5a6cd4349c144653be19a1ba4c254bbf626bdfd97c54d1c13ba63e396006eeac) = 210661183206a064208b3fc347287f904965c1ace6937ee230f8351bb3387d80 SHA256 (5c2445f99c567cf4c4521cd9ec8873f8788a4c7ce7a349dbc91d1bace373df80) = fb903771238383b516e78189fc382d48b8d8b2e21232ad8a87f02d039060ebfa SHA256 (5fe64c7db492eac1404a82eeb6f83341df3f3793ddb604e575e90d24a3de64dc) = fcb71b978e2ea06225abd3e725228910b9eeb52e432d331c88049bd7df053f69 SHA256 (7f1906900272125315833b9f0a6abc3c5af7b836d604fdb10a98a9079c1b99f9) = f4350476aa84805a2efe70c89849ad8ee1ad451c9678ddb28eda1306f686364a SHA256 (8f67a6dfad5be14636c80e4b4c4d7c42bd921009b33261b017fa54d07ad58910) = 8f67a6dfad5be14636c80e4b4c4d7c42bd921009b33261b017fa54d07ad58910 SHA256 (c77e17a2af1d2811f8afcdbc0faf7d6a3acc324730e4a36317e753d0fc9aa3fc) = f80171efe5b86e53df2a7254f9fa335ac1d2eca3c1ba9210bdd1cb50d6ffa4a5 SHA256 (d51a9833acf265d475408cde04e9beb75f950355580b9c73292ffbfbebf280eb) = a8dabbbae9352d4faa5ceb86a1ddf453d95dd3891c2ae4d52b56ecf8aa1de280 SHA256 (d8f42c764ad73c2370713d5f03ceb255bdd96518130a95955606c9ad511823bf) = d8f42c764ad73c2370713d5f03ceb255bdd96518130a95955606c9ad511823bf SHA256 (f5f24f0af60ff5b4a7d054e95bc677befbe708859fafdd5b214a47ed4d71d452) = 2e0fe3cc142b0926ab4143b3fd72566ae7b9ba9a3d96e3cdbc14966b58513561 ``` I'm lost where the issue is... since the "write to tar" should only happen when the checksum is correct -- so maybe the swapfs is confused / has somewhere an issue? As mentioned, this is with a single parallel download (which doesn't prevent that there can be multiple handles used at the swap at the same time).

hannes commented

2024-11-04 19:17:36 +00:00

Owner

Further debugging, the sha256 that we read out of swap is not the same that we write into it.

Doing some printf debugging in swap shows that the handle.blocks is only 0. And also the allocations / frees are sequential. So, I guess the append or read_partial has some minor issue (but can't focus anymore).

Further debugging, the sha256 that we read out of swap is not the same that we write into it. Doing some printf debugging in swap shows that the `handle.blocks` is only 0. And also the allocations / frees are sequential. So, I guess the append or read_partial has some minor issue (but can't focus anymore).

hannes commented

2024-11-05 10:07:17 +00:00

Owner

Disregard my last comment, there's (at least) some issue in our http client(s):

Disregard my last comment, there's (at least) some issue in our http client(s): - https://git.robur.coop/robur/http-mirage-client/pulls/2 - https://github.com/robur-coop/http-lwt-client/pull/26

reynir commented

2024-11-05 12:52:03 +00:00

Author

Owner

Nice! Thank you for your investigation and nice work. To me the code is now simpler and seem more robust (with the http-mirage-client behavior change).

hannes commented

2024-11-06 11:30:14 +00:00

Owner

with the http-mirage-client 0.0.8 that is fixed, this PR results in very nice archives where their checksums match \o/

hannes added 1 commit 2024-11-06 11:57:51 +00:00

organize failures into sections 37008e81f3

hannes added 1 commit 2024-11-06 12:08:35 +00:00

reset errors, and sort failures be87d19797

hannes commented

2024-11-06 12:10:32 +00:00

Owner

From my perspective, this is ready to merge. It may be worth to cut a mirage-swapf release first. It may as well be worth to investigate log messages and condense them (now that we have a status page).

hannes added 1 commit 2024-11-06 12:15:19 +00:00

use K.remote, not the entire repo c3d5c74075

hannes added 2 commits 2024-11-08 12:51:40 +00:00

remove logging from opam_file (now reported to status) 1e75be6900

log less, now that status is around 1f9e3e6e23