dev_appserver's handling of skip_files on linux doesn't handle directories correctly [77504616]

Assigned

Bug

Status Update

No update yet.

Description

be...@khanacademy.org

created issue #1

Apr 3, 2018 07:00PM

dev_appserver reads the application's skip_files setting, such that it avoids file-watching, and does not allow sandboxed code to access, any files that would not be uploaded to production. However, while appcfg/gcloud apply the skip_files rules to both files and directories [1], the inotify file-watcher does not consistently apply the rules to directories. In particular, it appears [2] to only match against top-level directories. That is, if I put `foo$` in my skip-files, it will successfully ignore a file called "foo", or a directory called "foo", but if I put `foo/bar$` it will only ignore a file foo/bar, not a directory foo/bar, whereas appcfg/gcloud would ignore both.

[1] In particular, google.appengine.tools.appcfg.FileIterator and googlecloudsdk.api_lib.app.util.FileIterator
[2] In particular, google.appengine.tools.devappserver2.inotify_file_watcher.InotifyFilewatcher._add_watch_for_path, currently around line 180 -- it looks at `self._skip_files_re` only if `dirpath in self._directories`, which is effectively only true for the root directory.

Comments

kn...@google.com <kn...@google.com> #2Apr 3, 2018 09:53PM

Assigned to kn...@google.com.

[Comment deleted]

be...@khanacademy.org <be...@khanacademy.org> #3Apr 3, 2018 09:55PM

Would you be able to provide an example app which produces an error for which gcloud doesn't report, but appcfg.py does?

kn...@google.com <kn...@google.com> #4Apr 3, 2018 11:11PM

In this case it is not app specific, The 400 error is due to quota of the application versions being over the limit which is 10 (but for this specific project it has been increased to 60). So any project with more than 10 versions will experience this.

be...@khanacademy.org <be...@khanacademy.org> #5Apr 3, 2018 11:31PM

[Comment deleted]

kn...@google.com <kn...@google.com> #6Apr 4, 2018 07:14PM

I have forwarded this request/report to the engineering team. We will update this issue with any progress updates and a resolution.

be...@khanacademy.org <be...@khanacademy.org> #7Apr 4, 2018 07:54PM

It's the same, but I'm happy to paste here for convenience

$ gcloud info --anonymize
Google Cloud SDK [196.0.0]

Platform: [Linux, x86_64] ('Linux', 'diffeomorphism', '4.13.0-37-generic', '#42-Ubuntu SMP Wed Mar 7 14:13:23 UTC 2018', 'x86_64', 'x86_64')
Python Version: [2.7.14 (default, Sep 23 2017, 22:06:14) [GCC 7.2.0]]
Python Location: [/usr/bin/python2]
Site Packages: [Disabled]

Installation Root: [${HOME}/google-cloud-sdk]
Additional Repositories:

https://storage.googleapis.com/gae-runtimes-gcloud/components-2.json
Installed Components:
kubectl: []
app-engine-python: [0.nodejs_eap.8]
core: [2018.03.30]
gcloud: []
app-engine-python-extras: [1.9.63]
beta: [2017.09.15]
gsutil: [4.30]
bq: [2.0.31]
alpha: [2017.09.15]
System PATH: [${HOME}/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:${HOME}/.bin:${HOME}/.go/bin]
Python PATH: [${HOME}/google-cloud-sdk/lib/third_party:${HOME}/google-cloud-sdk/lib:/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload]
Cloud SDK on PATH: [False]
Kubectl on PATH: [${HOME}/google-cloud-sdk/bin/kubectl]

Installation Properties: [${HOME}/google-cloud-sdk/properties]
User Config Directory: [${CLOUDSDK_CONFIG}]
Active Configuration Name: [default]
Active Configuration Path: [${CLOUDSDK_CONFIG}/configurations/config_default]

Account: [b..t@k..g]
Project: [None]

Current Properties:
[core]
account: [b..t@k..g]
disable_usage_reporting: [False]
[component_manager]
additional_repositories: [

https://storage.googleapis.com/gae-runtimes-gcloud/components-2.json]

Logs Directory: [${CLOUDSDK_CONFIG}/logs]
Last Log File: [${CLOUDSDK_CONFIG}/logs/2018.04.04/12.48.03.665704.log]

git: [git version 2.14.1]
ssh: [OpenSSH_7.5p1 Ubuntu-10ubuntu0.1, OpenSSL 1.0.2g 1 Mar 2016]

kn...@google.com <kn...@google.com> #8Apr 4, 2018 10:45PM

Reassigned to gc...@google.com.

I have reported the issue to the back-end team so that they may investigate it, however there is no ETA for the fix at this time. All further updates should occur in this report.