117 lines
5.1 KiB
Plaintext
117 lines
5.1 KiB
Plaintext
DDP:
|
|
files.py -> entry.py?
|
|
when I find a file, I really need to validate which dir it is in when
|
|
importing (rename subtest to SUBTEST) and the contents will end up in the
|
|
previous dir)
|
|
Also, should really set # files per sub dir -- think this will all need a
|
|
better import loop tracking of dir you are in
|
|
|
|
Should use a session per Thread, so maybe
|
|
sess=[]
|
|
sess[job.id]=Session()
|
|
etc
|
|
|
|
### DB
|
|
|
|
### BACKEND
|
|
* need a "batch" processing system that uses ionice to minimise load on mara and is threaded and used DB to interact with gunicorn'd pa
|
|
|
|
|
|
DONE: pa_jobs (threaded python add separate from photoassistant)
|
|
DONE: takes over files.py
|
|
has ai.py
|
|
needs broad jobs to:
|
|
DONE: find files in {import_dir & storage_dir}
|
|
calc thumbs/hashes { " }
|
|
run AI against { " }
|
|
move files from import_dir to appropriate sub_dir in storage_dir (list will come from pa web)
|
|
|
|
to keep html easy, can keep routes:
|
|
/jobs/import/ -> importlog.html
|
|
/jobs/ai/ -> importlog.html???
|
|
/jobs/move/ -> importlog.html???
|
|
|
|
but not sure if any job won't have passes, num files, etc. BUT if it ever does this is how we deal with it
|
|
|
|
on start-up:
|
|
DONE: set state: initialising
|
|
DONE: get settings, if no import_dir -> set state: Configuration Needed
|
|
check DB for any files
|
|
-> if some, set state: awaiting jobs
|
|
-> if none, set state: Find Files (run that code)
|
|
-> when job finished, then Import Files (import_dir only)
|
|
-> when job finished, the set state: awating jobs
|
|
|
|
DONE: implications, pa_job needs to know if it depends on another, e.g. do find before import (in example above)
|
|
DONE: pa web needs to show status for the job engine IN YOUR FACE when not at awaiting jobs yet, afterwards, maybe a simple jobs(<badge>) that is clickable on the gui?
|
|
|
|
|
|
PROPOSED CHANGES:
|
|
DIR 1<-to->M FILE
|
|
DIR -> path_prefix (move from file), num_files_in_dir
|
|
would stat it specifically, rather than each file when scan for new (so last_scan date moves here from a generic settings)
|
|
FILE -> fname, size, type (vid/img, etc.), hash, thumb, has_unidentified_face
|
|
?has_face?,
|
|
|
|
in FILE_PERSON_LINK add:
|
|
refimg, link to AI_scan
|
|
AI_SCAN:
|
|
date of scan
|
|
version of code?
|
|
settings used
|
|
|
|
NewJob should occur per path (or potentially all paths in import_dir), then you know #files for new non-scan jobs
|
|
if we make jobs be minimum, then ditch pass, and just use wait_for...
|
|
|
|
|
|
Jobs should be:
|
|
scan for files in DIR -> returns knows num_files in DIR
|
|
get thumbs for files (in DIR)
|
|
TODO: The 2 above lines are in GenerateFileData AND work on all import_dir paths at once, need to split this up (so our current setup would be 5 jobs (1 fail) on borric):
|
|
Job-1: Scan images_to_process -> success (num_files_1)
|
|
Job-2: Scan C: -> fail (report back to web)
|
|
Job-3: scan new_image_dir -> success (num_files_2)
|
|
Job-4 (wait on 1): Gen thumbs images_to_process (on num_files_1)
|
|
Job-5 (wait on 3): Gen thumbs new_image_dir (on num_files_2)
|
|
(worst case if a job waited on job-2, and 2 failed, then auto-fail it.)
|
|
|
|
process AI (<1 person>) for files (in DIR), e.g.
|
|
Job-7: scan 'cam' in images_to_process (num_files_1)
|
|
Job-8 (wait for 7): scan 'cam' in new_image_dir (num_files_2)
|
|
Job-9 scan 'dad' in images_to_process (num_files_1)
|
|
Job-10 (wait fo 9)scan 'dad' in new_image_dir (num_files_2)
|
|
etc.
|
|
this way we ditch passes
|
|
|
|
|
|
|
|
num jobs active, num jobs completed, lets bin them from the pa_job_manager table -> calculate them everytime (simple select count(1) from job where pa_job_state == "Completed")
|
|
FE does not really care what 'state' the job engine is in anyway, so maybe we bin that table, make it a local class to pa_job_manager?
|
|
|
|
When AI kicks in, it processes per person per DIR, only compares to an image if it has_unidentified_face
|
|
|
|
Admin
|
|
-> reset face_flag
|
|
|
|
AI
|
|
-> rescan <dir> for <person> --> can do this on new image only
|
|
(optimised to only scan has_unidentified_face)... BUT if you change
|
|
thresholds on AI, or we get a new/better one some day, then it can
|
|
all images with faces, or if we 'reset face_flag' rescan all images
|
|
|
|
|
|
TODO: Handle file deletions from file system (remove dangling DIR/FILE combos) -- also when moving to storage_dir, need to reset DIR, but keep FILE data
|
|
|
|
### UI
|
|
|
|
### AI
|
|
* store reference images (UI allows this now)
|
|
* check images
|
|
* allow for threshold/settings to be tweaked from the GUI
|
|
- it would be good to then say, just run the scanner against this image or maybe this DIR, to see how it IDs ppl
|
|
|
|
### SORTER
|
|
* date stuff
|
|
* exif processing?
|
|
* location stuff - test a new photo from my camera out
|