We were constantly having odd (we thought "un-reproduceable") problems with a GP Service while we were developing, deploying and maintaining it in production over the last year. We couldn't figure out why sometimes it worked, sometimes it didn't and (as far as we could tell), we hadn't really changed anything. Now, there was one item that we knew was problematic, but it didn't explain some of the things we were experiencing. Recently, we addressed that one item by rewriting some of the script and in doing so opened ourselves up to a host of problems along the way. In the end, they all seem to have come down to one aspect of all this that I thought I'd share in case it helps anyone else with their work.
It turns out that even the most (apparently) innocuous changes to a Python script written for a GP Service on ArcGIS Server can have a profound impact on the SD created and whether the Service can even be started after it's deployed to a Server or not. And, if it can, whether it's actually doing what you think it is at runtime. It turns out that the Python code you use to access a resource within the script itself impacts what gets bundled up in the Service Definition file, and consequently whether or not that SD file can be started up and used on an ArcGIS Server. Even if it works (depending on whether you include the data in the SD file or not and how you wrote the code), the service may not actually be doing what you think it is at runtime.
Our particular site has 2 Linux ArcGIS Servers that each have access to a shared location through a mount of the same name on both hosts. So, let's say, in our script we're querying "/mountname/somedir/anotherdir/file.gdb" to get a result that we display to the end user. We assumed, in doing something like this, that if we needed to update the information in the fgdb at some later time that, as long as the data model was the same (as in, we just needed to add some additional records), those should come up in the results the next time a query was passed. Not so...and it's because of a combination of the ArcPy script and the Service Defintion creation functionality. This assumes that you get that far of course...these same variables can result in your not even being able to start a service that (as far as a coder is concerned) hasn’t actually changed.
We learned that, while the following three examples of Python code have the exact same result in ArcMap when the code is being created/tested/etc (for development purposes, we created a "mountname" with "somedir" in it and "anotherdir" in that and placed "file.gdb" in it on our desktops to mimic the environment in Linux), they do not result in the same service definition content and do not run the same (if it all) when that SD is deployed to ArcGIS Server.
1.
strFgdbName = "file.gdb"
strInPath = os.path.normpath("/mountname/somedir/anotherdir")
self.strInPathFgdbName = os.path.join(strInPath, strFgdbName)
2.
strFgdbName = "file.gdb"
strInPath = os.path.normpath("/mountname/somedir")
self.strInPathFgdbName = os.path.join(strInPath, "anotherdir", strFgdbName)
3.
strFgdbName = "file.gdb"
strInPath = os.path.normpath(["/mountname/somedir/anotherdir"][0])
self.strInPathFgdbName = os.path.join(strInPath, strFgdbName)
It was a week ago that we figured this stuff out so my memory might be a bit off, but I believe it goes like this...
The first one will work whether you include the data in the service definition file or not. However, at runtime, it's not actually querying the fgdb at /mountname/somedir/anotherdir/file.gdb. It's actually querying a copy of that file that it has placed in the SD file (again, even if you don't include data when you create it - these SDs being created for an unknown server at the time of their generation, it should be noted, because we use the same SD files to deploy to various environments that are all architecturally the same but accessed via different URLs, etc) and extracted to its v101 directory content. So, if I added records to the shared location, they wouldn't be returned in any new queries because the service wasn't actually looking at that file geodatabase.
The second one can't even be started after it's deployed because it can't find file.gdb...of course, it's not actually looking where it should be either. And, here, it doesn't matter whether I include the data or not when I create the SD either.
If you actually look at the python script wrapped up in the SDs created by the above scripts, you can see (as we discovered) that it was actually changing the code that it was subsequently deploying to the ArcGIS Server and either breaking the service (as in the second) or making it do something unintended (as in the first).
Eventually, we stumbled on the third approach. It works, I assume, because the method of getting the path into the string is too complicated for whatever it is that is trying to help us out when we create the SD file. Consequently, it does nothing "helpful" and leaves the script alone; that is, leaves it as it was written. And then we can successfully deploy and run our service as we expect.
I'm sure that whatever it is doing when it's creating the SD files is being done for a reason but I've never stumbled across anything in the documentation that suggested it was changing scripts in a way that might have unforeseen results. So, I thought I'd just pass this along in case it helps someone else...
cheers,
jtm