Time/Date views in Notes can cause performance problems, yet they are a
very important way to display information to the user.
To appreciate the value of Time/Date views, think about the following
examples:
1 A view that displays only Severity One problems that have been entered
today.
2 A view that displays only orders placed within the last seven days.
3 A view that displays only technotes updated within the last 30 days.
4 A column in a mail file that displays either the time or the date a
message was sent. If the message was sent today, it displays just the
time; otherwise, it displays just the date.
Why do these views cause performance problems? The reason is that the
selection formula or column formula contains @functions such as @Today or
@Now and as such, the formula result is never static. Because of the
changing nature of the @function results, the index needs to be updated
every time the view is accessed. Unless you have a small set of data, this is
going to make the view slow to display.
Several techniques have been developed to allow time/date views to have
good performance, even if they contain a large set of data.
Creating Time/Date Views Using Selection Formulas
Here we describe three ways to create time/date views using selection
formulas, and we identify the pros and cons of each method.
Use @Today in the Selection Formula
This is the easiest solution and is always up to date, but it can have the worst
performance of all the methods. Try to avoid using this type of selection for
large views.
In this example we want to select all documents that have the field DueDate
within seven days of today’s date.
Selection formula
Select all documents with a DueDate within seven days of today.
SELECT @Adjust(DueDate; 0;0;7;0;0;0) > @Today
Advantages
• This kind of view will always be up to date.
• There are no other tasks to perform.
Disadvantages
• When you open the view the index must be rebuilt. If the number of
documents is more than a handful, there will be a noticeable delay,
perhaps many seconds for large views.
• The view refresh symbol will always display in the top left, indicating
(incorrectly) that the view is out of date. This will encourage the user to
click it (or press F9). Doing so will cause a similar delay to the one
experienced when the view was opened.
• Whenever UPDATE or UPDALL runs against the database, this view
will need to be completely rebuilt.
Use @Environment in the Selection Formula
This solution is also simple and has good performance, but requires an
environment variable to be kept up to date.
We use the same example as before: that is, we want to select all documents
that have the field DueDate within seven days of today’s date.
Selection formula
Select all documents with a DueDate within seven days of an environment
value.
SELECT @Adjust(DueDate; 0;0;7;0;0;0) > @Environment("Today")
Other tasks
• You must create a scheduled agent that runs every night, after midnight
to update the value of the environment variable, $Today. This agent
does not have to reside within the database that has the time/date view,
but the agent must run on every server that contains a replica copy of
the database. The formula for the agent can be as simple as:
@SetEnvironment("Today"; @Text(@Today));
• The view index must be rebuilt once per day, after the environment
variable has been set. This can be accomplished using:
UPDALL dbName -r -t viewName
You can create a program record to perform this view update.
Advantages
• There is no delay when opening the view.
• The view refresh symbol behaves normally. That is, the symbol appears
only if updates occur while the view is open.
• When UPDATE or UPDALL runs against the database, the view will
update very quickly.
Disadvantages
• If the agent fails to run, the view will be out of date.
• If you make a replica copy on a new server and forget to create an agent
on this server, the view will not function properly (in most cases, the
selection formula will return an error and the view will display nothing).
In fact, prior to the first time the agent runs, the view will not function
properly.
• If the program record fails to execute, the view will be out-of-date (until
either the server performs the view rebuild, or a user presses SHIFT+F9
while in the view).
• If you make a replica copy on a new server, and forget to create a
program record for that server, the view will correctly display the first
time it is used, but will thereafter be out-of-date (until either the server
performs the view rebuild, or a user presses SHIFT+F9 while in the
view).
Use @Text in the Selection Formula
This third solution is simple, has good performance, and needs no
environment variable.
Again, the same example is used here: we want to select all documents that
have the field DueDate within seven days of today’s date.
Selection formula
Select all documents that contain “Today” in the list of dates within seven
days of DueDate. Use date-to-text formatting that translates today into
“Today”.
ddt := @Date(DueDate);
dwk := ddt : @Adjust(ddt; 0;0;1;0;0;0) : @Adjust(ddt;
0;0;2;0;0;0) :
@Adjust(ddt 0;0;3;0;0;0) : @Adjust(ddt; 0;0;4;0;0;0) :
@Adjust(ddt; 0;0;5;0;0;0) : @Adjust(ddt; 0;0;6;0;0;0);
dwk_fmt := @Text(dwk; "T1S3");
SELECT @If (@Contains(dwk_fmt; "Today"); @True; @False);
Other tasks
• The view must be updated once per day, after midnight. This can be
accomplished using:
UPDALL dbName -r -t viewName
You can create a program record to perform this view update.
Advantages
• There is no delay when opening the view.
• The view refresh symbol behaves normally. That is, the symbol appears
only if updates occur while the view is open.
• When UPDATE or UPDALL runs against the database, the view will
update very quickly.
• You do not have to run an agent every night to update an environment
variable.
• The view will never simply “not function properly.” The worst case
scenario is that the view will not be up to date. When it is first created, it
will be up to date automatically.
Disadvantages
• If the program record fails to execute, or if you make a replica copy on a
new server and forget to create the program record for that server, then
the view will be out-of-date (until either the server performs the view
rebuild, or they press SHIFT+F9 while in the view).
Creating Time/Date Views Using Agents
This alternative to using a time/date selection formula relies on agents
making changes to the documents themselves, or moving the document into
or out of a folder.
Save a DisplayInView Indicator in the Document
This method relies on the fact that every document has a computed field
which will contain a specific value (mark) whenever the document is saved.
All documents with this value will appear in a view of recently modified
documents. A scheduled agent is then run (daily) to unmark documents
which are older than seven days, so that they no longer appear in the view.
Selection formula:
Select all documents with the “DisplayInView” mark set.
SELECT DisplayInView = “1”
Other tasks
• Code is required in the QuerySave event to initialize the DisplayInView
field to “1” during document save. In our example, the setting would be
dependent on the value of DueDate. If DueDate is within seven days of
today, we set the indicator; otherwise we clear it. This QuerySave code
must be in every form that creates documents which will appear in the
view.
• An agent must be run daily to update the DisplayInView field:
'*** Update all ‘DisplayInView’ indicators
Sub Initialize
Dim s As New NotesSession
Dim db As NotesDatabase
Set db = s.CurrentDatabase
Dim dc As NotesDocumentCollection
Dim datetime As New NotesDateTime( "" )
Set dc = db.Search( {@adjust(DueDate; 0;0;7;0;0;0) < _
@Today & DisplayInView = "1"}, datetime, 0 )
Dim doc As NotesDocument
Set doc = dc.GetFirstDocument
While Not doc Is Nothing
doc.DisplayInView = ""
Call doc.save( True, True )
Set doc = dc.GetNextDocument( doc )
Wend
End Sub
Advantages
• There is no delay when opening the view.
• The view refresh symbol behaves normally. That is, the symbol appears
only if updates occur while the view is open.
• When UPDATE or UPDALL runs against the database, the view will
update very quickly.
• There is no need for a program record in the Domino Directory.
Disadvantages
• If the agent is not run, the view will be out-of-date.
• If the database replicates, changes must propagate to other replicas
before the first users enter the database in the morning.
• The database search may be resource intensive for large databases. This
is particularly important if many views require time/date document
updates, either in this database or others on this server.
Put Documents into a Folder
Put documents into a Most Recent folder. Run a scheduled agent (daily) to
remove documents which are older than seven days.
Selection formula
Not applicable. Folders do not have selection formulas. Documents are
moved in and out of folders under program (or user) control.
Other tasks
• Code is required in the QuerySave event to put the document into the
Most Recent folder. In our example, the action would be dependent on
the value of DueDate. If DueDate is within seven days of today, we put
the document into the folder. If not, we remove it from the folder, if
required. This QuerySave code must be in every form that creates
documents which will appear in the folder.
• An agent must be run daily to remove old documents from the folder:
Sub Initialize
Dim s As New NotesSession
Dim db As NotesDatabase
Set db = s.CurrentDatabase
Dim view As NotesView
Set view = db.GetView( "Most Recent" )
Dim doc As NotesDocument
Set doc = view.GetFirstDocument
'*** Clean out the folder
Do While Not doc Is Nothing
doc.RemoveFromFolder( "Most Recent" )
Set doc = view.GetNextDocument( doc )
Loop
'*** Re-populate the folder
Dim dc As NotesDocumentCollection
Dim datetime As New NotesDateTime( "" )
Set dc = db.Search( {@adjust(DueDate; 0;0;7;0;0;0) > _
@Today}, datetime, 0 )
Call dc.PutAllInFolder( "Most Recent" )
End Sub
Advantages
• There is no delay when opening the folder.
• There is no need for a program record in the Domino Directory.
• There are no modifications made to documents (placing a document into
a folder updates the folder, not to the document).
Disadvantages
• If the nightly agent is not run, the folder will be out of date.
• If the database replicates, changes must propagate to other replicas
before the first users enter the database in the morning.
• The database search may consume significant CPU resources for large
databases. This is particularly important if many views require
time/date document updates, either in this database or others for this
server.
• Users may move documents into and out of the folder. There is no ACL
setting to prevent this activity.
Domino.Commands
Users interact with Domino applications through commands issued as
URLs. Each command specifies a top-level object and activates the methods
needed to carry out a request and generate HTML output. The Domino
Web Server implements many commands. The commands related to the
display and modification of documents are worth a closer look. These are
the commands that enable interactive applications, so the similarities and
differences in their implementation are of particular interest.
?OpenDocumentand ?EditDocument
Three commands display the contents of a document:
?OpenDocumentopens an existing document in read mode, ?EditDocument
opens an existing document in edit mode, and ?OpenForm opens a new
document in edit mode. The implementations of ?OpenDocument and
?EditDocument are very similar. Both execute the following sequence of
steps:
1. Open the existing document.
2. Read all items from document into memory.
3. Load the Notes form referenced by this document.
4. Scan the Notes form and execute default value formulas for any items
that are not yet in memory, and execute all computed field formulas
and add/update items to the in-memory document accordingly.
5. Run the appropriate agent if an item named $$QueryOpenAgent exists.
6. Render the Notes form into an HTML page or HTML form using the
item values from the in-memory document.
7. Free the in-memory document and all items it contains.
The only significant difference between ?OpenDocument and
?EditDocument commands is in step 6, where ?OpenDocument instructs the
GenerateHTML methods to respect read-mode hide attributes contained
within the Notes form, and to create HTML for a read-only page.
?EditDocument instructs these methods to respect edit-mode hide attributes
on the Notes form and to create an HTML form.
?OpenForm
The ?OpenFormcommand executes a similar sequence of steps:
1. Create a new document in memory.
2. Load the Notes form referenced in the URL.
3. Scan the Notes form and execute all default value formulas and
computed field formulas, adding items to the in-memory document.
4. Run the appropriate agent if an item named $$QueryOpenAgent exists.
5. Render the Notes form into an HTML form, respecting edit-mode hide
attributes and using item values from the in-memory document.
6. Free the in-memory document and all items it contains.
The last step of the procedure for all three commands frees the in-memory
document and its associated items. The ?EditDocument and ?OpenForm
commands do not cache the documents for later use because that would
violate the stateless nature of the a Web server. If and when the browser
sends edited information back, the Domino Web Server will re-establish all
the data structures necessary to handle the updates.
?CreateDocumentand ?SaveDocument
?CreateDocumentand ?SaveDocument are the two commands that receive
HTTP POST data generated by browsers when a user clicks the Submit
button on an HTML form and saves the data in Notes documents. Submit
buttons on HTML forms generated by the ?OpenForm command send the
?CreateDocument command, and Submit buttons on HTML forms
generated by the ?EditDocument command send the ?SaveDocument command. The two commands follow similar sequences of steps. The steps
for ?CreateDocument begin with:
1. Create a new document in-memory.
2. Open the Notes form referenced in the URL.
3. Scan the Notes form and execute all default field formulas and
computed value formulas, adding items to the in-memory document.
?SaveDocument
The steps for ?SaveDocument begin with:
1. Read the existing document referenced in the HTTP POST data.
2. Open the Notes form referenced in the URL.
3. Scan the Notes form and execute all default field formulas and
computed value formulas, adding items to the in-memory document.
Then both commands continue on essentially the same path:
4. Create items for all data sent in the POST data.
5. Scan the Notes form and execute all translation, validation and
computed value formulas, updating items in the in-memory document
and returning validation errors as HTML.
6. Scan the Notes form and eliminate any computed-for-display items
from the in-memory document.
7. Run the appropriate agent if an item named $$QuerySaveAgent exists.
8. Save the in-memory document to the appropriate database file.
9. If the $$Return item exists, run the appropriate formula from the Notes
form, and if there is no output from a QuerySave agent, send the result
of this formula back to the browser.
10. Delete the in-memory document and all associated items.
One thing worth noting is the fact that formulas that a user might have
thought executed only once at the time that an HTML form was sent by
either the ?EditDocument or ?OpenForm command will actually execute
again when ?CreateDocument or ?SaveDocument is processed. This can
result in some confusion if these formulas have time-dependent values, or
are based on lookups of constantly changing information.
This list of limits in Domino R5.0 and R4.6 has been compiled from Notes
Help and the Lotus Knowledge Base.
A couple of people responded to my performance tips paper with the suggestion that GetItemValue and ReplaceItemValue should be used in preference to just doc.fieldname.
I agree with the idea (though not enough to always do it myself!), but my testing suggests that performance is not the main reason to do it. I did a little test assigning a value to a field many times using the two different techniques. I found that ReplaceItemValue is a little faster -- 0.0000563 seconds versus 0.0000585 seconds. Not really enough difference to get excited about. Certainly, it's a very small amount of time compared to how long it takes to open a note.
The real reason for using GetItemValue and ReplaceItemValue, in my opinion, is to avoid maintenance problems with your code if IBM adds a new property or method that happens to have the same name as your field. Many people with fields named Lock had this problem when the Lock method was introduced, the next time they tried to edit their old code.
I don't really see the motivation for the extra typing in cases where your fieldname is so unique that the odds of a problem are vanishingly small, or so common that we would be silly to give a new thing the same name (e.g. I think we can be pretty sure there will not be a new property named Form, because it would break so many apps).
- Andre Guirard
Few more reasons :
1) You can pass in variables / constants for your field names. In other words, you can compute a field name based on a scenario in your code, and use that in the statement—you can’t do that with dot notation
2) If you make a habit of using constants / variables for field names, you avoid silly typos causing strange errors in your code: assuming you have Option Declare enabled, you will catch any errors in your field name references at compile time
3) Get/ReplaceItemValue makes it more readable to. You don't need to think "is this an actual form property or is it a field".
4) you can call doc.ReplaceItemValue( "fld", "val" ).IsReaders = True or set another item property.
5) There is definetly a performance penalty in using the dot notation which is invisible if you access 10 items but may have it's importance if you have an agent accessing 1000 items running 10 times currently from the web.
Domino provides many ways to lay out pages, using both HTML and Domino database
constructs such as forms and fields. This section focuses on approaches to laying out the appearance of a page using static elements. The next section summarizes ways to design forms for maximum performance.
Server Configuration
Here are few server configuration
The first step in examining or building a Web application is designing the architecture.
How many pages, views, and databases are involved? The table below contains
Suggestions that should be considered at this architectural level. Many of the suggestions revolve around a couple of major themes:
- Indexing views takes time and processing power
- Cached data is served faster than non-cached data
Note that some of these suggestions, although they directly affect application
performance, are often handled by the server administrator instead of the application
developer.
INTRODUCTION
A common experience with new developers, is that they create an application that works great for them, but after it's in use for a while, they discover that three factors have caused their application's performance to degrade to the point of near uselessness. These three factors are:
Increasing number of documents.
Increasing number of users.
Huge numbers of deleted documents.
However, this slowdown isn't inevitable. If you're careful designing your application, you can retain reasonable performance with large numbers of documents and users, and avoid excessive deletions.
DESIGN FOR PERFORMANCE
The following isn't everything there is to know about building high-performing applications. We've concentrated on issues likely to be encountered by new developers, and those that are most likely to cause problems that aren't evident during development and testing, but only emerge later on.
Note: we use the term "expensive" here to denote something that will degrade performance -- something that takes a long time to execute or calculate.
The more fields you put on your forms, the more they will slow your application – not only is the form itself slower, but it takes longer to index views, so the application is slower to open and navigate.
“Computed for display” fields only affect the performance of the forms they are on – not view performance. Use Computed for Display when possible instead of Computed fields.
The more documents your database stores, the longer it will take to index your views. Databases containing millions of documents are possible, but only if the greatest care is taken in the design will an application perform acceptably with more than a hundred thousand or so.
The more you modify documents, the longer it will take to index views. Each time a view is used, the index must be brought up to date with any documents that were created or modified since it was last used.
The more you modify documents, the longer it will take to replicate your application. This is particularly a problem when you have many users with local replicas, using your application offline. If your application is tying up their replication, their email is delayed, they can't finish up and disconnect from the network, etcetera.
Therefore, avoid modifying documents unnecessarily.
Deleting a document and then recreating “the same” document is three times as bad in terms of performance, as just modifying the existing document.
NOTE: One frequent situation we see, are Notes applications that load their data from some other source – a text file or relational database. Every night, an agent runs. The agent deletes every document in the database, then reads the outside data source and creates a new Notes document for each record. This is the worst possible thing you can do for performance. There are algorithms available to do this synchronization with the least necessary change to your documents, which you can easily find by searching on the dW Notes forums (see Resources appendix).
More views will slow your application even if those views aren't used much. Think about the tasks your users need to perform and what views will be useful. Delete useless views.
The more columns there are in a view, the longer it takes to index the view. Include only those columns users need.
The more complex your column formulas are, the longer it takes to index the view. Consider moving particularly complex formulas into a computed field on a form, so that your views can just refer to the field -- particularly if the same column is used in multiple views. There's a tradeoff here because having more fields also slows things, so exercise judgment.
A view with a column re-sort is less expensive in terms of performance than two separate views, but a re-sort is not free. Avoid unneeded re-sorts, and use the column option “Defer index creation until first use” (new in version 8.0) to speed up views that have re-sorts.
A full-text index can take up a lot of room on disk, but it's usually worth it in terms of performance. That's because if you don't have the index, users are forced to use slower searching methods that tie up the server and take longer to give them results.
NOTE: A new database property in version 8.0 lets you turn off "full-text" searching of a database with no index. This decreases the amount by which users can tie up the server, but does not eliminate this problem as there's another searching technique that causes similar issues.
It's possible to create views that display the same document several times. For instance, you may have a multi-valued “Category” field and you categorize the view by this field, making each document appear under each category listed in that document. This is often a very useful thing to do, and we're not saying not to do it. But do be aware that this sort of view is expensive, in direct ratio to how many times each document appears in it.
Reader fields are somewhat expensive, especially in cases where the view contains many documents, but very few to which the user does have access. Of course, sometimes you must use Readers fields, but be aware of the cost, and consider ways to limit the number of documents or avoid using views (e.g. by mailing a doclink to the user who needs to see a document).
Categorized views are a better-performing choice when you have reader fields, provided you don't use the view option "Don't display empty categories." [note: confirm wording of this option].
The use of @Now or @Today in the selection formula of a view, or in the column formulas of views or folders, is very expensive, and should be avoided if at all possible. Investigate alternatives, such as:
Just sort the view in reverse order by date, so that today's documents are at the top.
Use a view categorized by date, and use @SetViewInfo to restrict the display to today's documents only.
The same kind of categorized view, embedded in a page, with a “single category” formula.
Use a folder instead of a view. Run an agent overnight which finds all the documents that should be in the view, and places them there. Set the accesses of the folder so that users may not manually add and remove documents.
Don't use the GetNthDocument method to iterate through a collection of documents. GetFirstDocument and GetNextDocument are better for this.
Don't automatically use the “NoCache” option on every @DbLookup and @DbColumn command. When you do this in a keyword formula, it slows your form significantly, and most keyword lists don't change every day. The cost of NoCache every time a document is opened, often isn't worth the convenience of not having to close and reopen the application to see an updated list once in a great while. Think about whether you really have to have that data "up to the second."
If your keyword formula calls @DbColumn, then uses @Unique to remove the duplicates, this is a sign that you're probably doing it wrong. Instead, use a categorized view or one with "Unique keys for ODBC Access" [confirm name of option] turned on, so that the keyword values are already unique. Once you get many documents, @DbColumn can take a long time, but returning 20 category headings takes about the same amount of time no matter how many documents are in the categories.
For many types of list fields, their keyword lookups don't have to be done when the document is opened in read mode. Use a keyword formula such as the following to avoid the performance hit when users are just viewing a document. [insert formula here]
TESTING
Finally, a key technique for making sure your design performs well going forward, is to actually test it under as realistic conditions as you can manage.
Estimate how many documents the database will ultimately have in it, and create twice that many test documents. It's usually possible to create realistic test data by creating a single test document, and a formula agent to make copies of the document assigning random values to certain key fields. [insert an example of such an agent here].
Note: When testing the design of an application that's already in use, make sure to do your testing on a copy of the application -- not a replica -- so that there's no chance of your test documents replicating into the production environment. It's also best to use a dedicated testing server, in case your application does anything that crashes the server or ties up the its resources.
Bear in mind that accessing data on a server is slower than accessing data locally, so performance testing done with a local database doesn't reflect performance in actual use.
Bear in mind that end users' machines and network connections may not be as fast as yours. Test performance on a lower-end system with a poor connection to see what the real user experience will be like.
It may be difficult to simulate conditions of high user load if you are doing all the testing by yourself. There are partner products for this, and [any other suggestions?]...
curtsey : IBM Site
Posts
