ddp/photoassistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix BUG-64: can now move files into import or storage path

Browse Source
This commit is contained in:
Damien De Paoli
2021-09-26 21:14:08 +10:00
parent 25a50d029f
commit 23c8d16a5b
7 changed files with 119 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
BUGs
View File

@@ -3,5 +3,3 @@ BUG-56: when making a viewing list of AI:mich, (any search?) and going past the
BUG-60: entries per page (in folders view) ignores pagesize, and this also contributes to BUG-56 I think
BUG-61: in Fullscreen mode and next/prev occasionally dropped out of FS
this is just another consequence of going beyond Pagesize, when we get new entries from DB, it loses FS flag
BUG-64: seems when we move files, they get a new? FILE entry -- the file, hash, thumb, faces, etc. should all still be connected to the FILE entry and just make it have a new home / new location...
- this is both with a move via GUI, and theoretically also when we find a moved file on the FS (could use hash check to validate if we can just keep connections)

47
TODO
View File

@@ -1,15 +1,56 @@
## GENERAL
* close button on invalid password should look like danger/alert/close for jobs
* work out why no thumbs for:
pa=# select e.name from entry e, file f where f.eid = e.id and e.type_id =1 and f.thumbnail is null;
name
-------------------------------------------
dad and mum wedding with priest.bmp
dad with albina.bmp
dad portrait.bmp
presents.bmp
dad and mums wedding party.bmp
images.png
images (2).png
the boys.bmp
dad mum willie and emilio.bmp
dad with ross on trike.bmp
dad mum out.bmp
snowies machinery.bmp
images (1).png
dad and mums wedding party with names.bmp
dads wedding.bmp
dads wedding2.bmp
pa=# select e.name from entry e, file f where f.eid = e.id and e.type_id = 2 and f.thumbnail is null;
name
------------------------
20210526_205257_01.mp4
20210526_205257_99.mp4
2014-xmas-hps.xcf
rabbits.xcf
IMG_2553.MOV
IMG_2553.MOV
IMG_2553.MOV
DSCN0553.MOV
DSCN0555.MOV
DSCN0552.MOV
DSCN0556.MOV
DSCN0554.MOV
(including why .xcf is seen as a video???)
* remember last import dir, so you can just go straight back to it
* put a delete option on viewer page
* remember last import dir, so you can just go straight back to it
* close button on invalid password should look like danger/alert/close for jobs
* maybe strip unnecessary / at end of directory name. i think i have left behind empty folders, e.g. check switzerland and austria
- also should allow move to existing folder soon...
* move all unsorted photos/* -> import/
fix: BUG-64 first
-- should make a select with right-click also enable move/del buttons
TEST if this works with moving a folder/dir and a combo of folder/dir and a file
fix first: BUG-64 [DONE] and the folder move todo above <- TODO
* metadata at folder level with file level to add more richness

14
files.py
View File

@@ -1,6 +1,7 @@
from wtforms import SubmitField, StringField, HiddenField, validators, Form
from flask_wtf import FlaskForm
from flask import request, render_template, redirect, send_from_directory, url_for
from path import MovePathDetails
from main import db, app, ma
from sqlalchemy import Sequence
from sqlalchemy.exc import SQLAlchemyError
@@ -287,7 +288,8 @@ def files_ip():
OPT=Options( request )
entries=GetEntries( OPT )
people = Person.query.all()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT, people=people )
move_paths = MovePathDetails()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT, people=people, move_paths=move_paths )
################################################################################
# /files -> show thumbnail view of files from storage_path
@@ -298,7 +300,8 @@ def files_sp():
OPT=Options( request )
entries=GetEntries( OPT )
people = Person.query.all()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT, people=people )
move_paths = MovePathDetails()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT, people=people, move_paths=move_paths )
################################################################################
@@ -310,7 +313,8 @@ def files_rbp():
OPT=Options( request )
entries=GetEntries( OPT )
people = Person.query.all()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT )
move_paths = MovePathDetails()
return render_template("files.html", page_title=f"View Files ({OPT.path_type} Path)", entry_data=entries, OPT=OPT, move_paths=move_paths )
################################################################################
@@ -323,7 +327,8 @@ def search():
# always show flat results for search to start with
OPT.folders=False
entries=GetEntries( OPT )
return render_template("files.html", page_title='View Files', search_term=request.form['search_term'], entry_data=entries, OPT=OPT )
move_paths = MovePathDetails()
return render_template("files.html", page_title='View Files', search_term=request.form['search_term'], entry_data=entries, OPT=OPT, move_paths=move_paths )
################################################################################
# /files/scannow -> allows us to force a check for new files
@@ -448,6 +453,7 @@ def delete_files():
@app.route("/move_files", methods=["POST"])
@login_required
def move_files():
jex=[]
for el in request.form:
jex.append( JobExtra( name=f"{el}", value=request.form[el] ) )

26
internal/js/files_support.js
View File

@@ -29,9 +29,17 @@ function RunAIOnSeln(person)
$.ajax({ type: 'POST', data: post_data, url: '/run_ai_on', success: function(data){ window.location='/'; return false; } })
}
function change_rp_sel()
{
icon_url = $('option:selected', '#rp_sel').attr('icon_url')
$('#move_path_icon').html( '<svg id="move_path_icon" width="20" height="20" fill="currentColor"><use xlink:href="'
+ icon_url + '"></svg>' )
$('#move_path_type').val( $('option:selected', '#rp_sel').attr('path_type') )
}
// show the DBox for a move file, includes all thumbnails of selected files to move
// and a pre-populated folder to move them into, with text field to add a suffix
function MoveDBox(sps, db_url)
function MoveDBox(path_details, db_url)
{
$('#dbox-title').html('Move Selected File(s) to new directory in Storage Path')
div =`
@@ -39,27 +47,23 @@ function MoveDBox(sps, db_url)
<p class="col">Moving the following files?</p>
</div>
<form id="mv_fm" class="form form-control-inline col-12" method="POST" action="/move_files">
<input id="move_path_type" name="move_path_type" type="hidden"
`
div += ' value="' + path_details[0].type + '"></input>'
div+=GetSelnAsDiv()
yr=$('.highlight').first().attr('yr')
dt=$('.highlight').first().attr('date')
div+=`
<div class="input-group my-3">
<alert class="alert alert-primary my-auto py-1">
<svg width="20" height="20" fill="currentColor"><use xlink:href="
`
div+=db_url+'#db"/></svg>'
if( sps.length > 1 ) {
// NB: alert-primary here is a hack to get the bg the same color as the alert primary by
div+= '<select name="storage_rp" class="text-primary alert-primary py-1 border border-primary rounded">'
for(p of sps) {
div+= '<option>'+p+'</option>'
div+= '<svg id="move_path_icon" width="20" height="20" fill="currentColor"><use xlink:href="' + path_details[0].icon_url + '"></svg>'
div+= '<select id="rp_sel" name="rel_path" class="text-primary alert-primary py-1 border border-primary rounded" onChange="change_rp_sel()">'
for(p of path_details) {
div+= '<option path_type="'+p.type+'" icon_url="'+p.icon_url+'">'+p.path+'</option>'
}
div+= '</select>'
} else {
div+= '/'+sps[0]+'/'
div+= '<input type="hidden" name="storage_rp" value="' + sps[0] + '">'
}
div+=`
</alert>
<input id="prefix" type="text" name="prefix" class="text-primary text-right form-control"

18
pa_job_manager.py
View File

@@ -1014,11 +1014,13 @@ def MoveFileToNewFolderInStorage(job,move_me, dst_storage_path, dst_rel_path):
print( f"MoveFileToNewFolderInStorage: {move_me} to {dst_storage_path} in new? folder: {dst_storage_path}")
try:
dst_dir=dst_storage_path.path_prefix + '/' + dst_rel_path
if DEBUG:
print( f"would make dir: {dst_dir}" )
os.makedirs( dst_dir,mode=0o777, exist_ok=True )
src=move_me.FullPathOnFS()
dst=dst_dir + '/' + move_me.name
os.replace( src, dst )
if DEBUG:
print( f"would mv {src} {dst}" )
except Exception as e:
print( f"ERROR: Failed to move file to new location on filesystem, err: {e}")
@@ -1032,13 +1034,16 @@ def MoveFileToNewFolderInStorage(job,move_me, dst_storage_path, dst_rel_path):
part_rel_path=""
for dirname in dst_rel_path.split("/"):
part_rel_path += f"{dirname}"
if DEBUG:
print( f"Should make a Dir in the DB for {dirname} with parent: {parent_dir}, prp={part_rel_path} in storage path" )
new_dir=AddDir( job, dirname, parent_dir, part_rel_path, dst_storage_path )
parent_dir=new_dir
part_rel_path += "/"
if DEBUG:
print( f"now should change {move_me} in_dir to {new_dir} created above in {dst_storage_path}" )
move_me.in_dir = new_dir
move_me.in_path = dst_storage_path
if DEBUG:
print( f"DONE change of {move_me} in_dir to {new_dir} created above" )
AddLogForJob(job, f"{move_me.name} - (moved to {os.path.dirname(move_me.FullPathOnFS())})" )
return
@@ -1263,7 +1268,6 @@ def JobRunAIOn(job):
session.commit()
for jex in job.extra:
print( jex )
if 'eid-' in jex.name:
entry=session.query(Entry).get(jex.value)
if entry.type.name == 'Directory':
@@ -1570,8 +1574,10 @@ def JobMoveFiles(job):
JobProgressState( job, "In Progress" )
prefix=[jex.value for jex in job.extra if jex.name == "prefix"][0]
suffix=[jex.value for jex in job.extra if jex.name == "suffix"][0]
storage_rp=[jex.value for jex in job.extra if jex.name == "storage_rp"][0]
dst_storage_path = session.query(Path).filter(Path.path_prefix=='static/Storage/'+ storage_rp).first()
path_type=[jex.value for jex in job.extra if jex.name == "move_path_type"][0]
rel_path=[jex.value for jex in job.extra if jex.name == "rel_path"][0]
dst_storage_path = session.query(Path).filter(Path.path_prefix=='static/' + path_type + '/'+ rel_path).first()
for jex in job.extra:
if 'eid-' in jex.name:
move_me=session.query(Entry).join(File).filter(Entry.id==jex.value).first()
@@ -1736,19 +1742,13 @@ def FindBestFaceMatch( dist, threshold ):
####################################################################################################################################
def ProcessFaceMatches( job, dist, threshold, e, name ):
while True:
print( f"ProcessFaceMatches() - finding best match left with dist={dist}" )
which_r, which_f, which_fd = FindBestFaceMatch( dist, threshold )
print( f"seems that best match is r={which_r}, f={which_f}, with fd={which_fd}" )
if which_r != None:
print( f"okay, which_r is real, so we have a match" )
MatchRefimgToFace( which_r, which_f, which_fd )
AddLogForJob(job, f'WE MATCHED: {name[which_r]} with file: {e.name} - face distance of {which_fd}')
# remove this refimg completely, cant be 2 of this person matched
print( f"now remove this refimg from dist" )
del( dist[which_r] )
# remove this face id completely, this face cant be matched by someone else
print( f"now remove this face from dist (if it is connected with anyone else)" )
print( f"dist now = {dist}" )
RemoveFaceNumFromDist( dist, which_f )
else:
return

26
path.py
View File

@@ -1,3 +1,5 @@
from shared import PA, ICON
from flask import url_for
from flask_wtf import FlaskForm
from main import db, app, ma
from sqlalchemy import Sequence
@@ -31,13 +33,29 @@ class Path(db.Model):
def __repr__(self):
return f"<id: {self.id}, path_prefix: {self.path_prefix}, num_files={self.num_files}, type={self.type}>"
################################################################################
# helper function to find StoragePathNames - used in html for move DBox to show
# potential storage paths to move files into
# Class describing PathDeatil (quick connvenence class for MovePathDetails())
################################################################################
def StoragePathNames():
class PathDetail(PA):
def __init__(self,type,path):
self.type=type
self.path=path
self.icon_url=url_for('internal', filename='icons.svg') + '#' + ICON[self.type]
return
################################################################################
# helper function to find oath details for move destinations - used in html
# for move DBox to show potential storage paths to move files into
################################################################################
def MovePathDetails():
ret=[]
sps=Path.query.join(PathType).filter(PathType.name=='Storage').all()
for p in sps:
ret.append(p.path_prefix.replace('static/Storage/','') )
obj = PathDetail( type='Storage', path=p.path_prefix.replace('static/Storage/','') )
ret.append( obj )
ips=Path.query.join(PathType).filter(PathType.name=='Import').all()
for p in ips:
obj = PathDetail( type='Import', path=p.path_prefix.replace('static/Import/','') )
ret.append( obj )
return ret

13
templates/files.html
View File

@@ -4,6 +4,17 @@
<script src="{{ url_for( 'internal', filename='js/files_support.js')}}"></script>
<script src="{{ url_for( 'internal', filename='js/files_transform.js')}}"></script>
<script>
var move_paths=[]
{% for p in move_paths %}
p = new Object()
p.type = '{{p.type}}'
p.path = '{{p.path}}'
p.icon_url = '{{p.icon_url}}'
move_paths.push(p)
{% endfor %}
</script>
<div class="container-fluid">
<form id="main_form" method="POST">
<input type="hidden" name="cwd" id="cwd" value="{{OPT.cwd}}">
@@ -58,7 +69,7 @@
<button aria-label="next" id="next" {{nxt_disabled}} name="next" class="next sm-txt btn btn-outline-secondary">
<svg width="16" height="16" fill="currentColor"><use xlink:href="{{url_for('internal', filename='icons.svg')}}#next"/></svg>
</button>
<button aria-label="move" id="move" disabled name="move" class="sm-txt btn btn-outline-primary ms-4" onClick="MoveDBox({{StoragePathNames()|safe}},'{{url_for('internal', filename='icons.svg')}}'); return false;">
<button aria-label="move" id="move" disabled name="move" class="sm-txt btn btn-outline-primary ms-4" onClick="MoveDBox(move_paths,'{{url_for('internal', filename='icons.svg')}}'); return false;">
<svg width="16" height="16" fill="currentColor"><use xlink:href="{{url_for('internal', filename='icons.svg')}}#folder_plus"/></svg>
</button>
{% if "files_rbp" in request.url %}