Web Services Update "Screen Scraping"

Web services are presenting Java and NET developers new easy-to-learn opportunities to access and integrate with valuable legacy assets, according to a growing number of enterprise apps professionals. See why non-programmatic approaches, including even screen scraping could boost career opps for Java and .NET enterprise devs.

Tags: Mainframe, Integration, Web Services, Guruge, Legacy, Applications, Screen Scraping,

Web services could give new life to old-fashioned "screen scraping" styles of client-to-mainframe integration.

Anura Guruge, an independent web services and legacy apps consultant from Gilford, N.H., contends that after massive spending in the mainframe software sector in the years leading up to Y2K, there has been "apathy' in such spending, largely due to budget constraints.

But, he added, a new wave of web services-based approaches can be used to extend mainframe assets throughout the enterprise, which is is sparking new interest in IT pros that can deliver "mainframe modernization," he said.

The current climate, in fact, is presenting a unique opportunity for traditional web, Java and .NET developers and architects to add value to mainframe assets, without the need to learn COBOL, CORBA or other mainframe-centric technologies.

The Mainframe Integration Opps for Java, .NET
Guruge to suggest that a "non-programmatic" technique like screen scraping can be an advantage, not a handicap, to Java and .NET devs -- one they can wrap their arms -- and applications -- around. The trick for devs and architects that want to tap into valuable legacy data, applications and business rules, Guruge said, is to understand just how to construct a web service that can call to a legacy (proprietary) destination.

"The point is that you are developing new applications, but you are not recreating transactions that have been around for 20-plus years," said Guruge. To succeed at this style of mainframe Web services, integration developers must do four (4) key things:

*Identify the target mainframe transactions
*Set up a SOAP connection
*Call and retrieve the mainframe-side data, and
*Merge the results with other elements of a composite application

"Screen Scraping" Grows Up
'Screen scraping' has become a somewhat derogatory term for many enterprise IT professionals, mainly because it's been associated with more than a few integration debacles over the past decade or so. And, moreover, many programmers looked down at the idea of interacting with mainframes using calls that effectively emulated a terminal.

But Guruge insists that today's web services environment presents a whole new range of capabilities for using calls, and much richer opportunities for non legacy devs, including Java and .NET to use them to good advantage.

But, rather than call the approach 'screen scraping,' Guruge prefers to describe the approach as a 'runtime execution model.' The key is that the approach is non-programmatic.

A "Runtime Execution Model" Integration Approach
With REM, the dev isolates the applications he wants to invoke on the mainframe side, and then calls these applications into a "composite of applications" running in the environment the dev knows best, .NET or Java environments, for instance.

The integration is accomplished by the call to the mainframe app, which in effect becomes simply just another object in the dev's Web service.

The approach is enabled by use of a host integration server such as those offered by IBM, NetManage, WRQ, and others. When needed, the integration server steps through a mainframe access process - 'scraping' a succession of screens - and returns results. The legacy application continues to run unchanged on its native platform.

The runtime execution model (which also has been described as a "host integrator" model), was well-established before web services and XML came about. Now that a call to the mainframe app is now a web services call may give the approach better respectability, especially in an era when IT managers are looking for more standards-based ways to integrate with the mainframe, rather than replace it.

The types of legacy 'applications' Guruge recommends for "run time execution" are not those with clean separation of data and business logic. These can be handled by data integrators and app servers.

Instead the apps considered are apps full of CICS-like transactions intermingling data and logic. This code has gone through many iterations in a long life, and is in turn difficult to identify. "Y2K proved conclusively that finding source code is not as easy as you think," he said.

Guruge said a key to REM's quick deployment arises from the introduction of what he calls "Host Integration Studio tools" that help developers capture the necessary mainframe log-in and authentication access procedures, as well as the general screen navigation [screen scraping] to access needed transactions.

The tools, which may increasingly hook into framework standards such as Eclipse, also help you describe the system I/O in terms of WSDL. Capturing of I/O screens occur almost as with Microsoft Office application development, where macros are run and recorded. Too, WSDL descriptions are built with the help from wizards.

As there are quite a number of things running in this execution model, what about performance overhead? The mainframe is not greatly affected, according to Guruge. But the Web services engines cannot be underpowered. "You may not want low-end Pentiums running your host integration," Guruge said dryly. "Platforms in the middle must deliver [according to] mission-critical criteria.," he said.

"This is a mature paradigm with advantages," he continued, "what was driven in the past from the terminal is now run by an exterior application."

One common mistake made by IT professionals doing legacy integration is not looking at the wide picture, Guruge said. . "People get blindsided, [by] not looking at everything that is out there. In fact, they do not have to restrict themselves to a single scheme," he said. "Start with something small, and manageable, but representative,"

Bad integration design can derail any project, but better designs may be more widely obtainable with defined, standard Web services implementations, Guruge suggested. "With things like CORBA - which is still a very powerful technology - there was complexity. With Web services you are getting standards infrastructure - XML, SOAP, WSDL. And you simplify the invocation."

"Web services are simpler, at least in terms of a usage model. I think it is an easier to understand and realize model," he said. Screen scraping should not be derided, said Guruge. "It works, it's a proven technique, it tends to be simple, and it also allows you to combine transactions from multiple applications," he added.

"In terms of drawbacks, well, here we are talking about continuing with legacy applications. With this technique legacy applications never go away," said Guruge. "It's been said before: Big Iron platforms are not dead. "For the big volumes and workload the mainframes still tend to be the ideal platform."