e57d880c44
sql: use $N instead of ?N as numbered placeholder
2021-11-18 11:40:15 +01:00
675b57a579
Rename Builder_db modules in bin/ and test/
...
They both depend on the Builder_db module in db/, and using the same
name confuses Merlin (as well as myself sometimes).
2021-11-15 15:12:16 +01:00
Robur
b43c6f4d79
Refactor default_datadir into library builder_system
2021-11-12 13:22:43 +00:00
977678b325
Detect datadir by platform
2021-11-12 13:04:51 +00:00
Robur
e69d1beb9f
Add platform to build table
2021-11-05 10:45:26 +00:00
a9ff2dd033
Remove rresult
2021-11-05 10:10:44 +00:00
4a42cffc6c
verify-data-dir: change level for untracked files
...
Untracked files are not necessarily an error so log at warning level
instead.
2021-09-14 16:03:12 +02:00
70e240e7b0
verify-data-dir: check for untracked files
2021-09-14 14:45:27 +02:00
71a016fdae
Fix up mix up
...
When builds were uploaded the script and console was mixed up. This
commit fixes that and includes a fixup migration script.
2021-09-10 13:00:04 +02:00
Robur
dd6535296d
use Cstruct.length instead of deprecated Cstruct.len
2021-09-08 09:11:54 +00:00
Robur
8279bc1c26
store relative path for console and script
2021-09-07 11:53:51 +00:00
Robur
0628938898
verify-data-dir: check build script & console as well
2021-09-07 10:04:54 +00:00
Robur
f7823da321
fix console output order
2021-09-07 10:04:11 +00:00
Robur
0efcec460d
fix
2021-09-07 09:35:26 +00:00
Robur
4126cab805
Fix migration M20210712c
2021-09-07 09:35:26 +00:00
Robur
94feffdcc2
Fixup builder-migrations: add M20210712c
2021-09-07 09:35:26 +00:00
3fe6e83300
M20210712c: fix rollback
2021-09-07 09:35:26 +00:00
10f78877e9
M20210712c: fix indentation for clarity
2021-09-07 09:35:26 +00:00
Robur
17420c389b
WIP
2021-09-07 09:35:26 +00:00
Robur
0d918192ea
builder_db.exe: verify_data_dir verifies job name and uuid as well
2021-09-07 09:22:25 +00:00
Robur
392286dd98
remove README.md and build-hashes from build_artifact table and file system
...
no longer store full file on disk
2021-07-12 14:10:54 +00:00
Robur
928821fec6
fix migration 2021-07-12
2021-07-12 13:42:13 +00:00
Robur
9195c91ab5
remove result_kind from build table (de-duplicate information), add some indexes
2021-07-12 13:31:36 +00:00
e8f918230f
verify-data-dir-stream ( #50 )
...
builder-db verify-data-dir: stream build artifacts
Co-authored-by: Reynir Björnsson <reynir@reynir.dk>
Reviewed-on: https://git.robur.io/robur/builder-web/pulls/50
Co-Authored-By: hannes <hannes@mehnert.org>
Co-Committed-By: hannes <hannes@mehnert.org>
2021-07-12 10:42:03 +00:00
Robur
7c4bf56da6
builder_db: add a verify-data-dir subcommand
...
This subcommand checks that all files in the build_artifact table
are present in the data directory. Also, their size and hash must
match.
2021-07-07 13:33:26 +00:00
Robur
b09001916b
fixup 2021-07-07d: remove initial ./ from fpath
2021-07-07 13:00:57 +00:00
Robur
1b4b27e1c5
fixup 2021-07-07c: strip .hvt/.xen images if not already stripped
...
updates build_artifact and also the main_binary in build
2021-07-07 12:36:30 +00:00
Robur
49f7502e0c
fixup 2021-07-07b: move *.deb.debug to bin/*.deb, update build_artifact and build (main_binary)
2021-07-07 11:29:10 +00:00
Robur
f66fa8bf19
fixup 2021-07-07a: remove leftover orb.deb / orb.txz from build_artifacts
2021-07-07 10:50:12 +00:00
Robur
96ee7649b7
fix 0706 migration (old build table should be named new_build)
2021-07-07 10:45:14 +00:00
Robur
6ec40365ab
Add input_id to build table
...
The input_id column is a checksum of the files relevant for reproducibility.
2021-07-06 13:54:25 +00:00
7c7282894b
Typed database IDs ( #47 )
...
Typed database IDs
Reviewed-on: https://git.robur.io/robur/builder-web/pulls/47
Co-Authored-By: Reynir Björnsson <reynir@reynir.dk>
Co-Committed-By: Reynir Björnsson <reynir@reynir.dk>
2021-07-05 12:45:08 +00:00
cc092ca9d8
M20210701: reapply index
2021-07-05 10:23:09 +02:00
21065c9f44
Use initially deferred foreign key constraint
...
On build.main_binary.
2021-07-01 11:02:41 +02:00
37e68f91f4
Handle jobs without successful build in migrations
...
The migrations migrate-2021-06-29 and migrate-2021-06-30 would not apply
if a job exists without any successful build. Now the migrations script
silently skips jobs without succesful builds.
2021-07-01 10:56:14 +02:00
Robur
1e3fcf984f
Allow README.md being present anywhere
...
The one in root takes precedence, but opam-installer doesn't really
want to install things into %{prefix}%.
2021-06-30 14:10:02 +00:00
Robur
009fa49e9e
render README from latest build (preserved as a tag value in job_tag table)
2021-06-30 12:47:30 +00:00
Robur
eb786088e7
Builder_db.Job_tag.add: use INSERT OR REPLACE to avoid constraint violation
2021-06-30 09:56:47 +00:00
Robur
9a271add7b
add tag to jobs: description
2021-06-29 15:15:16 +00:00
Robur
e45497e97c
add tags to jobs: section and synopsis, inferred from the latest successful build
2021-06-29 14:59:08 +00:00
a3f9e9aba0
Add job-remove command to builder-db
...
`builder-db job-remove job-name` removes a job from builder-web
including its associated files.
2021-06-29 08:56:18 +00:00
Robur
b279eb521b
Adapt to build input files removal
...
* Do not synthesize empty input files list
* Drop table build_file
2021-06-25 10:26:03 +00:00
bde3baec46
Refactor migrations and don't enable foreign keys
...
Each migration is, for the most part, a module that exposes expected
database version numbers, command identifier and documentation. This
results in all information about the migration and rollback are
found in the module itself, and builder_migrations.ml only has to
reference the module.
Some migrations require foreign keys constraints are disabled. It is not
possible to enable or disable foreign key constraints inside a
transaction.
2021-06-10 12:08:14 +02:00
Robur
f5d6475d2d
fix migration, password_hash and salt are blobs
2021-06-09 13:59:04 +00:00
Robur
5cece213c2
Add a user-disable command, enforce foreign key constraints in builder_db.exe and migrations
2021-06-09 13:45:22 +00:00
Robur
31a501e0c4
Add user column to build
...
We want to track who uploaded what builds.
A `nologin` user is created in the migration that owns all preexisting builds.
2021-06-09 11:54:24 +00:00
Robur
02bfbc956f
Improve HTTP status codes (4xx on user data failure)
...
Fixes #43
2021-06-09 09:48:51 +00:00
Robur
eaf8a609c9
Add an access_list to grant per-job user upload permissions
...
Fixes #36
2021-06-08 14:54:23 +00:00
Robur
d986d614a8
Add http status metrics middleware
2021-06-08 12:36:53 +00:00
8f4a45bf76
Revise error handling
...
- also validate UUID to be 36 characters
- also error with not_found for unknown jobs
2021-06-08 10:26:57 +00:00
Robur
4fec5ed38c
Builder_db: use foreign key for build.main_binary
2021-06-02 10:29:08 +00:00
Robur
49003ca21f
Store relative paths for build artifacts in database
2021-06-01 15:43:55 +00:00
Robur
10351c65bd
Migrate to dream
2021-06-01 14:06:36 +00:00
User Builder
af1ed4cbc7
ignore sigpipe
2021-06-01 08:38:25 +00:00
User Builder
8a8dd9a14b
Add influx metrics reporting
2021-06-01 08:38:25 +00:00
User Builder
f7eafc56c5
prefix log lines with timestamp
2021-06-01 08:38:24 +00:00
fbe8748c02
Fix --datadir
...
It was erroneously specified as ----datadir
2021-05-18 17:02:45 +02:00
759172c4a7
Add --datadir parameter
2021-05-17 11:52:52 +02:00
561a81a951
Alter db build index
...
We started looking up by date a lot.
2021-04-29 15:16:19 +02:00
52da2cf588
Refactor list result iter
2021-03-12 09:47:28 +01:00
e2a489a74d
Refactor migrations
2021-03-10 10:51:34 +01:00
3b81c52c59
Expose database repr, database fixup
...
Parts of the database representation are exposed.
A fixup command for builder-migrations is added to remove bad database
entries fixed in a57798f4c0
2021-03-08 17:01:00 +01:00
3c33d66104
Add rollback of filesize column
2021-02-24 15:07:15 +01:00
9d4a77abd4
Do sqlite dance
2021-02-24 11:31:31 +01:00
535d2ac0b9
Add file sizes
2021-02-24 11:31:31 +01:00
7b81d78554
Use scrypt ( #32 )
...
Switch to using scrypt for password hashing
Co-authored-by: Reynir Björnsson <reynir@reynir.dk>
Reviewed-on: https://git.robur.io/robur/builder-web/pulls/32
Co-Authored-By: reynir <reynir@reynir.dk>
Co-Committed-By: reynir <reynir@reynir.dk>
2021-02-23 15:20:18 +00:00
8d211dc831
Check database version
...
Exit if the application id and user version are not as expected.
2021-02-17 14:15:57 +01:00
03fe06be87
Add index job_build_idx on build
...
This speeds up loading / by ~10x
2021-02-02 10:25:19 +01:00
56737ec71b
Migration code, model aware of main binary
...
Sqlite3 application_id and user_version are now set to identify the
database is a builder-web database, and the user_version represents the
schema version.
The 'build' table is extended with a 'main_binary' column. This
represents the main binary artifact from the build. This is decided by
there being exactly one file in bin/.
A migration tool is written that does both migrations and rollbacks, and
migration and rollback is implemented for the above mentioned change.
2021-01-29 10:15:31 +01:00
810e1393ed
Remove builder-db add command
...
Instead, use e.g.
find /var/db/builder/ -maxdepth 3 -name full -exec curl http://user:password@builder-web:3000/upload --data-binary @\{} \;
This will add the build to builder-web.
2021-01-27 20:10:27 +00:00
6f9a74ec65
Use dash instead of underscore in binary name
2021-01-22 11:12:49 +01:00
af505853e6
Database file is in /var/db/builder-web/
2021-01-22 10:59:03 +01:00
0f98541feb
Store files on disk
2021-01-21 17:51:58 +01:00
3a0d9222d5
Fixup user-update using user_add not user_update
2021-01-21 12:13:32 +01:00
a6fe9ada45
Add password_iter option
2021-01-21 12:01:47 +01:00
01babd0d0d
User authentication and upload endpoint
2021-01-21 10:11:37 +01:00
5298108d1a
Implement sqlite database backed builder-web
2021-01-19 10:44:32 +01:00
cd1cdcc9bb
Implement meta data caching
...
Also print builds by their start time instead of UUID
2020-12-07 15:43:54 +01:00
41b0b6c031
Implement job, job run
2020-12-07 10:17:49 +01:00
0a92bdae65
Initial commit
2020-12-04 18:10:21 +01:00
