Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
maarchRM
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Redmine
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Maarch
maarchRM
Commits
ceba7208
Verified
Commit
ceba7208
authored
5 years ago
by
Alexandre Morin
Browse files
Options
Downloads
Patches
Plain Diff
feat (#11785) : merge branch feature/compressedStorage
parent
42cee766
Branches
feat/11785_compressed_storage
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
dependency/repository/Adapter/fileSystem/Repository.php
+150
-71
150 additions, 71 deletions
dependency/repository/Adapter/fileSystem/Repository.php
with
150 additions
and
71 deletions
dependency/repository/Adapter/fileSystem/Repository.php
+
150
−
71
View file @
ceba7208
...
@@ -63,12 +63,11 @@ class Repository
...
@@ -63,12 +63,11 @@ class Repository
}
}
}
}
// CONTAINER
/**
/**
* Delete a container
* Delete a container
* @param string $name The name of container
* @param string $name The name of container
* @param mixed $metadata The object or array of metadata
* @param mixed $metadata The object or array of metadata
*
*
* @return mixed The address/uri/identifier of created container on repository
* @return mixed The address/uri/identifier of created container on repository
*/
*/
public
function
createContainer
(
$name
,
$metadata
=
null
)
public
function
createContainer
(
$name
,
$metadata
=
null
)
...
@@ -90,34 +89,34 @@ class Repository
...
@@ -90,34 +89,34 @@ class Repository
* Update a container metadata
* Update a container metadata
* @param string $name The name of container
* @param string $name The name of container
* @param mixed $metadata The object or array of metadata
* @param mixed $metadata The object or array of metadata
*
*
* @return bool
* @return bool
*/
*/
public
function
updateContainer
(
$name
,
$metadata
)
public
function
updateContainer
(
$name
,
$metadata
)
{
{
}
}
/**
/**
* Read a container metadata
* Read a container metadata
* @param string $name The name of container
* @param string $name The name of container
*
*
* @return mixed The object or array of metadata if available
* @return mixed The object or array of metadata if available
*/
*/
public
function
readContainer
(
$name
)
public
function
readContainer
(
$name
)
{
{
}
}
/**
/**
* Delete a container
* Delete a container
* @param string $name The name of container
* @param string $name The name of container
*
*
* @return bool
* @return bool
*/
*/
public
function
deleteContainer
(
$name
)
public
function
deleteContainer
(
$name
)
{
{
}
}
// OBJECTS
// OBJECTS
...
@@ -142,21 +141,21 @@ class Repository
...
@@ -142,21 +141,21 @@ class Repository
* Get a resource in repository
* Get a resource in repository
* @param mixed $path The path/uri/identifier of stored resource on repository
* @param mixed $path The path/uri/identifier of stored resource on repository
* @param integer $mode A bitmask of what to read 0=nothing - only touch | 1=data | 2=metadata | 3 data+metadata
* @param integer $mode A bitmask of what to read 0=nothing - only touch | 1=data | 2=metadata | 3 data+metadata
*
*
* @return mixed The contents of resource
* @return mixed The contents of resource
*/
*/
public
function
readObject
(
$path
,
$mode
=
1
)
public
function
readObject
(
$path
,
$mode
=
1
)
{
{
switch
(
$mode
)
{
switch
(
$mode
)
{
case
0
:
case
0
:
return
$this
->
checkFile
(
$path
);
return
$this
->
checkFile
(
$path
);
case
1
:
case
1
:
return
$this
->
readFile
(
$path
);
return
$this
->
readFile
(
$path
);
case
2
:
case
2
:
return
$this
->
readFile
(
$path
.
'.metadata'
);
return
$this
->
readFile
(
$path
.
'.metadata'
);
case
3
:
case
3
:
$data
=
array
();
$data
=
array
();
$data
[
0
]
=
$this
->
readFile
(
$path
);
$data
[
0
]
=
$this
->
readFile
(
$path
);
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
...
@@ -167,17 +166,16 @@ class Repository
...
@@ -167,17 +166,16 @@ class Repository
return
$data
;
return
$data
;
default
:
default
:
throw
new
\Exception
(
"This mode '
$mode
' isn't avalaible"
);
throw
new
\Exception
(
"This mode '
$mode
' isn't avalaible"
);
}
}
}
}
/**
/**
* Update a resource
* Update a resource
* @param string $path The URI of the resource
* @param string $path The URI of the resource
* @param string $data The content
* @param string $data The content
* @param object $metadata The new metadata to update or insert
* @param object $metadata The new metadata to update or insert
*
*
* @return bool
* @return bool
*/
*/
...
@@ -186,7 +184,7 @@ class Repository
...
@@ -186,7 +184,7 @@ class Repository
if
(
!
is_null
(
$data
))
{
if
(
!
is_null
(
$data
))
{
$this
->
updateFile
(
$path
,
$data
);
$this
->
updateFile
(
$path
,
$data
);
}
}
if
(
!
is_null
(
$metadata
))
{
if
(
!
is_null
(
$metadata
))
{
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
$this
->
updateFile
(
$path
.
'.metadata'
,
json_encode
(
$metadata
,
\JSON_PRETTY_PRINT
));
$this
->
updateFile
(
$path
.
'.metadata'
,
json_encode
(
$metadata
,
\JSON_PRETTY_PRINT
));
...
@@ -198,7 +196,7 @@ class Repository
...
@@ -198,7 +196,7 @@ class Repository
return
true
;
return
true
;
}
}
/**
/**
* Delete a resource
* Delete a resource
* @param string $path The URI of the resource
* @param string $path The URI of the resource
*
*
...
@@ -207,11 +205,10 @@ class Repository
...
@@ -207,11 +205,10 @@ class Repository
public
function
deleteObject
(
$path
)
public
function
deleteObject
(
$path
)
{
{
$this
->
deleteFile
(
$path
);
$this
->
deleteFile
(
$path
);
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
if
(
$this
->
checkFile
(
$path
.
".metadata"
))
{
$this
->
deleteFile
(
$path
.
".metadata"
);
$this
->
deleteFile
(
$path
.
".metadata"
);
}
}
return
true
;
return
true
;
}
}
...
@@ -221,7 +218,7 @@ class Repository
...
@@ -221,7 +218,7 @@ class Repository
/**
/**
* Get the directory to store
* Get the directory to store
* @param string $pattern The name or pattern for the collection
* @param string $pattern The name or pattern for the collection
*
*
* @return string The diretory name
* @return string The diretory name
*/
*/
protected
function
getDir
(
$pattern
)
protected
function
getDir
(
$pattern
)
...
@@ -237,7 +234,7 @@ class Repository
...
@@ -237,7 +234,7 @@ class Repository
if
(
!
is_dir
(
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
))
{
if
(
!
is_dir
(
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
))
{
mkdir
(
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
,
0775
,
true
);
mkdir
(
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
,
0775
,
true
);
}
}
}
}
return
$dir
;
return
$dir
;
...
@@ -307,9 +304,7 @@ class Repository
...
@@ -307,9 +304,7 @@ class Repository
return
$package
;
return
$package
;
}
}
}
}
}
}
$package
=
str_pad
(
'1'
,
8
,
"0"
,
STR_PAD_LEFT
);
$package
=
str_pad
(
'1'
,
8
,
"0"
,
STR_PAD_LEFT
);
//mkdir($this->root . DIRECTORY_SEPARATOR . $dir . DIRECTORY_SEPARATOR . $package, 0775, true);
//mkdir($this->root . DIRECTORY_SEPARATOR . $dir . DIRECTORY_SEPARATOR . $package, 0775, true);
...
@@ -324,7 +319,7 @@ class Repository
...
@@ -324,7 +319,7 @@ class Repository
// Sanitize name
// Sanitize name
$name
=
basename
(
$path
);
$name
=
basename
(
$path
);
$name
=
$this
->
getName
(
$name
,
$dir
);
$name
=
$this
->
getName
(
$name
,
$dir
);
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
.
DIRECTORY_SEPARATOR
.
$name
;
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$dir
.
DIRECTORY_SEPARATOR
.
$name
;
if
(
!
$fp
=
fopen
(
$filename
,
'x'
))
{
if
(
!
$fp
=
fopen
(
$filename
,
'x'
))
{
...
@@ -349,7 +344,7 @@ class Repository
...
@@ -349,7 +344,7 @@ class Repository
}
}
throw
new
\Exception
(
"Error writing at path
$path
."
);
throw
new
\Exception
(
"Error writing at path
$path
."
);
}
}
return
$dir
.
DIRECTORY_SEPARATOR
.
$name
;
return
$dir
.
DIRECTORY_SEPARATOR
.
$name
;
}
}
...
@@ -358,18 +353,28 @@ class Repository
...
@@ -358,18 +353,28 @@ class Repository
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
if
(
!
file_exists
(
$filename
))
{
if
(
!
file_exists
(
$filename
))
{
return
false
;
return
$this
->
checkCompressedFile
(
$path
)
;
}
}
return
true
;
return
true
;
}
}
protected
function
checkCompressedFile
(
$path
)
{
list
(
$zipfile
,
$filename
)
=
$this
->
getCompressedPathinfo
(
$path
);
$zip
=
\laabs
::
newService
(
'dependency/fileSystem/plugins/zip'
);
$compressedFiles
=
$zip
->
contents
(
$zipfile
);
return
in_array
(
$filename
,
$compressedFiles
);
}
protected
function
readFile
(
$path
)
protected
function
readFile
(
$path
)
{
{
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
if
(
!
file_exists
(
$filename
))
{
if
(
!
file_exists
(
$filename
))
{
throw
new
\Exception
(
"Can not find resource at path
$path
"
);
return
$this
->
readCompressedFile
(
$path
);
}
}
if
(
!
$data
=
file_get_contents
(
$filename
))
{
if
(
!
$data
=
file_get_contents
(
$filename
))
{
...
@@ -379,6 +384,51 @@ class Repository
...
@@ -379,6 +384,51 @@ class Repository
return
$data
;
return
$data
;
}
}
protected
function
readCompressedFile
(
$path
)
{
list
(
$zipfile
,
$filename
)
=
$this
->
getCompressedPathinfo
(
$path
);
$tmpdir
=
\laabs
::
getTmpDir
()
.
DIRECTORY_SEPARATOR
.
str_replace
(
','
,
'.'
,
microtime
(
true
))
.
'.'
.
mt_rand
();
$tmpfile
=
$tmpdir
.
DIRECTORY_SEPARATOR
.
basename
(
$filename
);
$zip
=
\laabs
::
newService
(
'dependency/fileSystem/plugins/zip'
);
$zip
->
extract
(
$zipfile
,
$tmpdir
,
$filename
,
null
,
'e'
);
if
(
!
$data
=
file_get_contents
(
$tmpfile
))
{
throw
new
\Exception
(
"Can not read at path
$path
"
);
}
unlink
(
$tmpfile
);
return
$data
;
}
protected
function
getCompressedPathinfo
(
$path
)
{
$steps
=
\laabs\explode
(
DIRECTORY_SEPARATOR
,
$path
);
$filename
=
$this
->
root
;
while
(
$step
=
array_shift
(
$steps
))
{
$filename
.
=
DIRECTORY_SEPARATOR
.
$step
;
switch
(
true
)
{
case
is_dir
(
$filename
)
:
continue
;
case
is_file
(
$filename
.
'.7z'
)
:
return
[
$filename
.
'.7z'
,
$step
.
DIRECTORY_SEPARATOR
.
implode
(
DIRECTORY_SEPARATOR
,
$steps
)];
case
is_file
(
$filename
.
'.zip'
)
:
return
[
$filename
.
'.zip'
,
$step
.
DIRECTORY_SEPARATOR
.
implode
(
DIRECTORY_SEPARATOR
,
$steps
)];
default
:
throw
new
\Exception
(
"Can not find resource at path
$path
"
);
}
}
throw
new
\Exception
(
"Can not find resource at path
$path
"
);
}
protected
function
updateFile
(
$path
,
$data
)
protected
function
updateFile
(
$path
,
$data
)
{
{
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
...
@@ -400,7 +450,12 @@ class Repository
...
@@ -400,7 +450,12 @@ class Repository
if
(
!
$wl
=
fwrite
(
$fp
,
$data
))
{
if
(
!
$wl
=
fwrite
(
$fp
,
$data
))
{
fclose
(
$fp
);
fclose
(
$fp
);
if
(
!
unlink
(
$filename
))
{
if
(
!
unlink
(
$filename
))
{
throw
new
\Exception
(
"Impossible to write at path
$path
- The backup "
.
$path
.
".save"
.
" is available"
);
throw
new
\Exception
(
"Impossible to write at path
$path
- The backup "
.
$path
.
".save"
.
" is available"
);
}
}
rename
(
$backup
,
$filename
);
rename
(
$backup
,
$filename
);
...
@@ -410,9 +465,14 @@ class Repository
...
@@ -410,9 +465,14 @@ class Repository
if
(
$wl
!=
strlen
(
$data
))
{
if
(
$wl
!=
strlen
(
$data
))
{
fclose
(
$fp
);
fclose
(
$fp
);
if
(
!
unlink
(
$filename
))
{
if
(
!
unlink
(
$filename
))
{
throw
new
\Exception
(
"Error in writing the file
$path
and it's impossible to delete but a backup "
.
$path
.
".save"
.
" is available"
);
throw
new
\Exception
(
"Error in writing the file
$path
and it's impossible to delete but a backup "
.
$path
.
".save"
.
" is available"
);
}
}
rename
(
$backup
,
$filename
);
rename
(
$backup
,
$filename
);
throw
new
\Exception
(
"Error in writing the file. The file doesn't have a modification"
);
throw
new
\Exception
(
"Error in writing the file. The file doesn't have a modification"
);
}
}
...
@@ -420,11 +480,18 @@ class Repository
...
@@ -420,11 +480,18 @@ class Repository
if
(
!
hash
(
'md5'
,
$data
)
!=
hash_file
(
'md5'
,
$filename
))
{
if
(
!
hash
(
'md5'
,
$data
)
!=
hash_file
(
'md5'
,
$filename
))
{
fclose
(
$fp
);
fclose
(
$fp
);
if
(
!
unlink
(
$filename
))
{
if
(
!
unlink
(
$filename
))
{
throw
new
\Exception
(
"Error, the hash of data isn't equals at file hash and it's impossible to delete but a backup "
.
$path
.
".save"
.
" is avaialble"
);
throw
new
\Exception
(
"Error, the hash of data isn't equals at file hash and it's impossible to delete but a backup "
.
$path
.
".save"
.
" is avaialble"
);
}
}
rename
(
$backup
,
$filename
);
rename
(
$backup
,
$filename
);
throw
new
\Exception
(
"Error, the hash of data isn't equals at file hash. The file doesn't have a modification"
);
throw
new
\Exception
(
"Error, the hash of data isn't equals at file hash. The file doesn't have a modification"
);
}
}
if
(
!
unlink
(
$backup
))
{
if
(
!
unlink
(
$backup
))
{
...
@@ -435,49 +502,16 @@ class Repository
...
@@ -435,49 +502,16 @@ class Repository
}
}
private
function
deleteFile
(
$path
)
private
function
deleteFile
(
$path
)
{
{
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
$filename
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
$path
;
if
(
!
file_exists
(
$filename
))
{
if
(
!
file_exists
(
$filename
))
{
throw
new
\Exception
(
"No resource found at path
$path
"
);
return
$this
->
deleteCompressedFile
(
$path
);
}
}
if
(
$this
->
shred
)
{
if
(
$this
->
shred
)
{
$fp
=
fopen
(
$filename
,
'w'
);
$this
->
shred
(
$filename
);
$fileSize
=
filesize
(
$filename
);
$pass
=
0
;
//while ($pass != 6) {
$randNum
=
rand
(
0
,
255
);
$compNum
=
255
-
$randNum
;
$lastNum
=
-
1
;
do
{
$lastNum
=
rand
(
0
,
255
);
}
while
(
$lastNum
==
$compNum
);
// First pass with random num
$randContents
=
str_repeat
(
chr
(
$randNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$randContents
);
// Second pass with complement of random num
$compContents
=
str_repeat
(
chr
(
$compNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$compContents
);
// Third pass with a new random num
$lastContents
=
str_repeat
(
chr
(
$lastNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$lastContents
);
$pass
++
;
//}
fclose
(
$fp
);
}
}
if
(
!
unlink
(
$filename
))
{
if
(
!
unlink
(
$filename
))
{
throw
new
\Exception
(
"Can not delete at path
$path
"
);
throw
new
\Exception
(
"Can not delete at path
$path
"
);
...
@@ -486,6 +520,52 @@ class Repository
...
@@ -486,6 +520,52 @@ class Repository
return
true
;
return
true
;
}
}
protected
function
deleteCompressedFile
(
$path
)
{
list
(
$zipfile
,
$filename
)
=
$this
->
getCompressedPathinfo
(
$path
);
$zip
=
\laabs
::
newService
(
'dependency/fileSystem/plugins/zip'
);
return
$zip
->
delete
(
$zipfile
,
$filename
);
}
protected
function
shred
(
$filename
)
{
$fp
=
fopen
(
$filename
,
'w'
);
$fileSize
=
filesize
(
$filename
);
$pass
=
0
;
//while ($pass != 6) {
$randNum
=
rand
(
0
,
255
);
$compNum
=
255
-
$randNum
;
$lastNum
=
-
1
;
do
{
$lastNum
=
rand
(
0
,
255
);
}
while
(
$lastNum
==
$compNum
);
// First pass with random num
$randContents
=
str_repeat
(
chr
(
$randNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$randContents
);
// Second pass with complement of random num
$compContents
=
str_repeat
(
chr
(
$compNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$compContents
);
// Third pass with a new random num
$lastContents
=
str_repeat
(
chr
(
$lastNum
),
$fileSize
);
fseek
(
$fp
,
0
);
fwrite
(
$fp
,
$lastContents
);
$pass
++
;
//}
fclose
(
$fp
);
}
protected
function
writeMetadata
(
$path
,
$metadata
)
protected
function
writeMetadata
(
$path
,
$metadata
)
{
{
$dir
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
dirname
(
$path
);
$dir
=
$this
->
root
.
DIRECTORY_SEPARATOR
.
dirname
(
$path
);
...
@@ -551,7 +631,6 @@ class Repository
...
@@ -551,7 +631,6 @@ class Repository
}
}
$jsonMetadata
=
$path
.
'='
.
json_encode
(
$metadata
,
\JSON_PRETTY_PRINT
)
.
"
\n
"
;
$jsonMetadata
=
$path
.
'='
.
json_encode
(
$metadata
,
\JSON_PRETTY_PRINT
)
.
"
\n
"
;
$mdh
=
fopen
(
$mdf
,
"r+"
);
$mdh
=
fopen
(
$mdf
,
"r+"
);
$mdl
=
false
;
$mdl
=
false
;
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment