fixed issue where we cant do for loop over os.walk more than once, so now convert it to a list, and then go through that once to get overal_file_cnt, and then commit this to the job and Dir structure so that it is known on the joblog page, e.g. we get the progress bar on the first import once the os.walk is finished. Also cleaned the code to remove unnecessary string manipulation around symlink name in loop, it cant happen there. Finally updated TODO to remove this todo item (progress bar on first import) and a few other old TODOs, added new section on scanning storage_dir, the next big thing...

TODO

@@ -8,18 +8,16 @@
             (file_refimg_link --> file_refimg_link needs a face_num?)
 
 ### BACKEND
+    scan storage_dir
+        * need to be smart not to 'rescan' import_dir if it is inside storage_dir
+        * need to find / remove duplicate files from inside storage_dir and itself, and in import_dir and in storage_dir
+        * so we can AI that and search all photos)
+
     -- started on some basic optimisations (commit logs every 100 logs, not each log)
         - with debugs: import = 04:11, getfiledetails== 0:35:35
         - without debugs: import == 04:03, getfiledetails == 0:35:36 -- not a sig diff
         - with exifread & debug: import == 04:26
 
-    * CheckForDups():
-        in files.py 
-            -> need to process the form and ACT on it (by deleting files)
-
-    * try again with walk to go through loop once quickly just to add up files,
-    * then start the import dir counting up / progress
-
     *** Need to use thread-safe sessions per Thread, half-assed version did not work
 
     need a manual button to restart it in the GUI, 
@@ -43,8 +41,6 @@
         need some way to multiselect images [DONE]
         and then get them into a new "folder"
 
-        Will also need a 'real storage path' 
-            one day allow scanning this too if we want (so we can AI that and search all photos)
 
         timelineview? (I think maybe sunburst for large amounts of files, then maybe something more timeline-series for drilling in?)
             (vertical timeline, date has thumbnails (small) horizontally along
@@ -60,12 +56,7 @@
 
 
 ### SORTER
-    * duplicate files - this sql finds them:
-        select d1.path_prefix, e1.name, f1.hash, d2.path_prefix, e2.name, f2.hash from entry e1, file f1, dir d1, entry_dir_link edl1, entry e2, file f2, dir d2, entry_dir_link edl2 where e1.id = f1.eid and e2.id = f2.eid and d1.eid = edl1.dir_eid and edl1.entry_id = e1.id and edl2.dir_eid = d2.eid and edl2.entry_id = e2.id  and f1.hash = f2.hash and e1.id != e2.id order by f1.hash, f2.hash;
-
-	* date stuff
 	* exif processing?
 		* location stuff - test a new photo from my camera out
         -- image is in dir, need to look at exifread output
 
-    * also need to find / remove duplicate files from inside import_dir and in storage_dir