DDP:
    Should use a session per Thread, so maybe
        sess={}
        sess[job.id]=Session()
        etc
    Then when we finish 1 job, we need to force 'wake-up' / run any dependant jobs

    move scan now and force scan over to new file structures
        - scan now is over, but job-id's should be hrefs... need to last_import_date=0 for all Dirs, then call ProcessImportDirs for force scan
    optimise gen hash / thumbs to only process dirs with new content

### DB	

### BACKEND
	* need a "batch" processing system that uses ionice to minimise load on mara and is threaded and used DB to interact with gunicorn'd pa

    * pa_job_manager, needs ai code
    * needs broad jobs to:
        DONE: find files in {import_dir & storage_dir}
        DONE: calc thumbs/hashes { " }
        run AI against { " }
        move files from import_dir to appropriate sub_dir in storage_dir (list will come from pa web)

        NEW_FILE -> add, has_unidentified_face 
            ?has_face?, 

        AI_SCAN:
            id
            date of scan
            version of code?
            settings used
        AI_SCAN_FILE_LINK
            id to link to  AI_scan
            refimg used/found

    NewJob should occur per path (or potentially all paths in import_dir), then you know #files for new non-scan jobs
        if we make jobs be minimum, then ditch pass, and just use wait_for...


    Jobs should be:
        scan for files in DIR -> returns knows num_files in DIR
        get thumbs for files (in DIR)
            TODO: The 2 above lines are in GenerateFileData AND work on all import_dir paths at once, need to split this up (so our current setup would be 5 jobs (1 fail) on borric):
                Job-1: Scan images_to_process -> success (num_files_1)
                Job-2: Scan C: -> fail (report back to web)
                Job-3: scan new_image_dir -> success (num_files_2)
                Job-4 (wait on 1): Gen thumbs images_to_process (on num_files_1)
                Job-5 (wait on 3): Gen thumbs new_image_dir (on num_files_2)
                (worst case if a job waited on job-2, and 2 failed, then auto-fail it.)

        process AI (<1 person>) for files (in DIR), e.g.
            Job-7: scan 'cam' in images_to_process (num_files_1)
            Job-8 (wait for 7): scan 'cam' in new_image_dir (num_files_2)
            Job-9 scan 'dad' in images_to_process (num_files_1)
            Job-10 (wait fo 9)scan 'dad' in new_image_dir (num_files_2)
            etc.  
                this way we ditch passes
            


    num jobs active, num jobs completed, lets bin them from the pa_job_manager table -> calculate them everytime (simple select count(1) from job where pa_job_state == "Completed")
    FE does not really care what 'state' the job engine is in anyway, so maybe we bin that table, make it a local class to pa_job_manager?

    When AI kicks in, it processes per person per DIR, only compares to an image if it has_unidentified_face

    Admin
        -> reset face_flag

    AI
        -> rescan <dir> for <person> --> can do this on new image only
        (optimised to only scan has_unidentified_face)... BUT if you change
        thresholds on AI, or we get a new/better one some day, then it can
        all images with faces, or if we 'reset face_flag' rescan all images


    TODO: Handle file deletions from file system (remove dangling DIR/FILE combos) -- also when moving to storage_dir, need to reset DIR, but keep FILE data

### UI

### AI
	* store reference images (UI allows this now)
	* check images
    * allow for threshold/settings to be tweaked from the GUI
        - it would be good to then say, just run the scanner against this image or maybe this DIR, to see how it IDs ppl

### SORTER
	* date stuff
	* exif processing?
		* location stuff - test a new photo from my camera out