85 lines
3.4 KiB
Plaintext
85 lines
3.4 KiB
Plaintext
DDP:
|
|
Should use a session per Thread, so maybe
|
|
sess={}
|
|
sess[job.id]=Session()
|
|
etc
|
|
|
|
### DB
|
|
|
|
### BACKEND
|
|
* need a "batch" processing system that uses ionice to minimise load on mara and is threaded and used DB to interact with gunicorn'd pa
|
|
|
|
* pa_job_manager, needs ai code
|
|
* needs broad jobs to:
|
|
DONE: find files in {import_dir & storage_dir}
|
|
DONE: calc thumbs/hashes { " }
|
|
run AI against { " }
|
|
move files from import_dir to appropriate sub_dir in storage_dir (list will come from pa web)
|
|
|
|
NEW_FILE -> add, has_unidentified_face
|
|
?has_face?,
|
|
|
|
AI_SCAN:
|
|
id
|
|
date of scan
|
|
version of code?
|
|
settings used
|
|
AI_SCAN_FILE_LINK
|
|
id to link to AI_scan
|
|
refimg used/found
|
|
|
|
NewJob should occur per path (or potentially all paths in import_dir), then you know #files for new non-scan jobs
|
|
if we make jobs be minimum, then ditch pass, and just use wait_for...
|
|
|
|
|
|
Jobs should be:
|
|
scan for files in DIR -> returns knows num_files in DIR
|
|
get thumbs for files (in DIR)
|
|
TODO: The 2 above lines are in GenerateFileData AND work on all import_dir paths at once, need to split this up (so our current setup would be 5 jobs (1 fail) on borric):
|
|
Job-1: Scan images_to_process -> success (num_files_1)
|
|
Job-2: Scan C: -> fail (report back to web)
|
|
Job-3: scan new_image_dir -> success (num_files_2)
|
|
Job-4 (wait on 1): Gen thumbs images_to_process (on num_files_1)
|
|
Job-5 (wait on 3): Gen thumbs new_image_dir (on num_files_2)
|
|
(worst case if a job waited on job-2, and 2 failed, then auto-fail it.)
|
|
|
|
process AI (<1 person>) for files (in DIR), e.g.
|
|
Job-7: scan 'cam' in images_to_process (num_files_1)
|
|
Job-8 (wait for 7): scan 'cam' in new_image_dir (num_files_2)
|
|
Job-9 scan 'dad' in images_to_process (num_files_1)
|
|
Job-10 (wait fo 9)scan 'dad' in new_image_dir (num_files_2)
|
|
etc.
|
|
this way we ditch passes
|
|
|
|
|
|
|
|
num jobs active, num jobs completed, lets bin them from the pa_job_manager table -> calculate them everytime (simple select count(1) from job where pa_job_state == "Completed")
|
|
FE does not really care what 'state' the job engine is in anyway, so maybe we bin that table, make it a local class to pa_job_manager?
|
|
|
|
When AI kicks in, it processes per person per DIR, only compares to an image if it has_unidentified_face
|
|
|
|
Admin
|
|
-> reset face_flag
|
|
|
|
AI
|
|
-> rescan <dir> for <person> --> can do this on new image only
|
|
(optimised to only scan has_unidentified_face)... BUT if you change
|
|
thresholds on AI, or we get a new/better one some day, then it can
|
|
all images with faces, or if we 'reset face_flag' rescan all images
|
|
|
|
|
|
TODO: Handle file deletions from file system (remove dangling DIR/FILE combos) -- also when moving to storage_dir, need to reset DIR, but keep FILE data
|
|
|
|
### UI
|
|
|
|
### AI
|
|
* store reference images (UI allows this now)
|
|
* check images
|
|
* allow for threshold/settings to be tweaked from the GUI
|
|
- it would be good to then say, just run the scanner against this image or maybe this DIR, to see how it IDs ppl
|
|
|
|
### SORTER
|
|
* date stuff
|
|
* exif processing?
|
|
* location stuff - test a new photo from my camera out
|